Adaptive generation of a high dynamic range image of a scene, on the basis of a plurality of images obtained by non-destructive reading of an image sensor

Abstract

High dynamic range (HDR) images are generated on the basis of a plurality of images obtained by non-destructive reading of an image sensor, called NDRO images. An HDR image generation method includes: the determination of a criterion of desired quality for the HDR image; at least two non-destructive readings of the sensor delivering at least two successive NDRO images; the selection, as a function of the criterion of desired quality, of the first and of the last NDRO image to be used to generate the HDR image; the generation of the HDR image on the basis of information extracted from a series of successive NDRO images starting with the first and terminating with the last NDRO image to be used; the storage of a single image at one and the same time throughout the entire HDR generation phase.

Claims

1. A method for generating a high dynamic range image of a scene, from a plurality of images of said scene obtained by non-destructive readout of an image sensor, the plurality of images being non-destructive read out (NDRO) images, said sensor including a plurality of pixels arranged in matrix form, and each associated with a photoelectric conversion element enabling converting a received light into electric charges and accumulating the electric charges during an exposure time to the light, the method comprising: a first non-destructive reading out of said image sensor, delivering a first current NDRO image with index n=0, and storing of said first image, within a memory zone, in association with an exposure time associated with said first image; for each pixel of said first image, comparing a signal value corresponding to said electric charges accumulated by said pixel with a signal selection threshold, and selecting said pixel when said signal value associated with said pixel is above said signal selection threshold; at least one iteration of the following successive steps, for integer n, n varying from 1 to N: non-destructively reading out said image sensor, delivering a current NDRO image with index n, updating said memory zone by storing, for each pixel of said current NDRO image with index n not previously selected in an NDRO image preceding said current NDRO image, a signal value associated with said pixel, in combination with an exposure time associated with said current NDRO image with index n, and for each pixel of said current NDRO image with index n, comparing a signal value corresponding to said electric charges accumulated by said pixel with said signal selection threshold, and selecting said pixel when said signal value associated with said pixel is above said signal selection threshold; and generating said high dynamic range image from the stored signal values, weighted by said associated exposure times of said signal values associated with said selected pixels and of said exposure times stored in association.

2. The method for generating a high dynamic range image according to claim 1, wherein the method implements a storage of a single image at a given moment.

3. The method for generating a high dynamic range image according to claim 1, further comprising determining a desired quality criterion for said high dynamic range image and a maximum exposure time for said sensor.

4. The method for generating a high dynamic range image according to claim 3, wherein said desired quality criterion is a signal-to-noise ratio of the pixels of said high dynamic range image.

5. The method for generating a high dynamic range image according to claim 3, wherein said NDRO image with index N is: the first of said successive NDRO images in which all of the pixels are associated with a signal value greater than or equal to said signal selection threshold, or an NDRO image, associated with said maximum exposure time of said sensor, when the NDRO image contains at least one pixel associated with a signal value below said signal selection threshold.

6. The method for generating a high dynamic range image according to claim 1, wherein said generating said high dynamic range image comprises calculating a signal value associated with each pixel of said high dynamic range image by evaluated weight, based on the response from the sensor, of said signal value stored in said memory area for each of said pixels by said associated exposure time.

7. The method for generating a high dynamic range image according to claim 1, further comprising determining a minimum electric charge accumulation time (tmin) on the image sensor before the first non-destructive readout of said image sensor, and wherein said minimum accumulation time tmin satisfies the condition: $\frac{t_{\min }+\left(t_{\mathrm{NDRO}}\right)}{t_{\min }}<\frac{S_{\mathrm{sat}}}{S_0}$ where: t.sub.NDRO is a time between two of said successive iterations of non-destructive readout of said image sensor, S.sub.sat is a saturation signal of said image sensor, and S.sub.0 is the weakest usable electronic signal, except noise, coming from said image sensor.

8. A non-transitory computer program product on a recording medium, comprising program code instructions for carrying out the method according to claim 1, when it is executed by a processor.

9. A non-transitory recording medium on which a computer program is recorded comprising program code instructions for carrying out steps of the method for generating a high dynamic range image according to claim 1, when executed by a processor.

10. A system for generating a high dynamic range image of a scene, from a plurality of images of said scene obtained by non-destructive readout of an image sensor, the plurality of images being non-destructive read out (NDRO) images, said sensor including a plurality of pixels arranged in matrix form, and each associated with a photoelectric conversion element enabling converting a received light into electric charges and to accumulate the electric charges during an exposure time to the light, the system comprising: a computing unit configured to execute the method for generating a high dynamic range image according to claim 1.

11. A system for generating a high dynamic range image of the scene, from a plurality of images of said scene obtained by non-destructive readout of an image sensor, the plurality of images being non-destructive read out (NDRO) images, said system comprising: a sensor comprising a plurality of pixels arranged in matrix form, and each associated with a photoelectric conversion element enabling to convert a received light into electric charges and to accumulate the electric charges during an exposure time to the light, said sensor being adapted to operate in a non-destructive readout mode; one or more processors configured to compare, for each pixel of an NDRO image, among a plurality of successive NDRO images delivered by said sensor, a signal value corresponding to said electric charges accumulated by said pixel to a signal selection threshold, and to select said pixel when said signal value associated with said pixel is above said signal selection threshold and when said pixel has not been previously selected from a preceding NDRO image from said plurality of NDRO images; and a memory configured to store signal values associated with said selected pixels of said successive NDRO images in association with exposure times associated with said NDRO images, wherein the one or more processors is configured to generate said high dynamic range image from said signal values associated with said selected pixels and said exposure times stored in association.

12. The system for generating a high dynamic range image of a scene according to claim 11, further comprising an interface module for entry by a user of a desired quality criterion for said high dynamic range image and of a maximum exposure time of said sensor.

13. The method for generating a high dynamic range image according to claim 2, further comprising determining a desired quality criterion for said high dynamic range image and a maximum exposure time for said sensor.

14. The method for generating a high dynamic range image according to claim 4, wherein said NDRO image with index N is: the first of said successive NDRO images in which all of the pixels are associated with a signal value greater than or equal to said signal selection threshold, or an NDRO image, associated with said maximum exposure time of said sensor, when the NDRO image contains at least one pixel associated with a signal value below said signal selection threshold.

15. The method for generating a high dynamic range image according to claim 2, wherein said generating said high dynamic range image comprises calculating a signal value associated with each pixel of said high dynamic range image by evaluated weight, based on the response from the sensor, of said signal value stored in said memory area for each of said pixels by said associated exposure time.

16. The method for generating a high dynamic range image according to claim 3, wherein said generating said high dynamic range image comprises calculating a signal value associated with each pixel of said high dynamic range image by evaluated weight, based on the response from the sensor, of said signal value stored in said memory area for each of said pixels by said associated exposure time.

17. The method for generating a high dynamic range image according to claim 4, wherein said generating said high dynamic range image comprises calculating a signal value associated with each pixel of said high dynamic range image by evaluated weight, based on the response from the sensor, of said signal value stored in said memory area for each of said pixels by said associated exposure time.

18. The method for generating a high dynamic range image according to claim 5, wherein said generating said high dynamic range image comprises calculating a signal value associated with each pixel of said high dynamic range image by evaluated weight, based on the response from the sensor, of said signal value stored in said memory area for each of said pixels by said associated exposure time.

19. The method for generating a high dynamic range image according to claim 2, further comprising determining a minimum electric charge accumulation time (tmin) on the image sensor before the first non-destructive readout of said image sensor, and wherein said minimum accumulation time tmin satisfies the condition: $\frac{t_{\min }+\left(t_{\mathrm{NDRO}}\right)}{t_{\min }}<\frac{S_{\mathrm{sat}}}{S_0}$ where: t.sub.NDRO is a time between two of said successive iterations of non-destructive readout of said image sensor, S.sub.sat is a saturation signal of said image sensor, and S.sub.0 is the weakest usable electronic signal, except noise, coming from said image sensor.

20. The method for generating a high dynamic range image according to claim 3, further comprising determining a minimum electric charge accumulation time (tmin) on the image sensor before the first non-destructive readout of said image sensor, and wherein said minimum accumulation time tmin satisfies the condition: $\frac{t_{\min }+\left(t_{\mathrm{NDRO}}\right)}{t_{\min }}<\frac{S_{\mathrm{sat}}}{S_0}$ where: t.sub.NDRO is a time between two of said successive iterations of non-destructive readout of said image sensor, S.sub.sat is a saturation signal of said image sensor, and S.sub.0 is the weakest usable electronic signal, except noise, coming from said image sensor.

Description

4. LIST OF FIGURES

(1) Other aims, features and advantages of the invention will appear more clearly upon reading the following description, provided as a simple illustrative and non-limiting example, in connection with figures, in which:

(2) FIG. 1 is a flowchart illustrating one embodiment of an adaptive method for generating an HDR image from a plurality of NDRO images obtained by non-destructive readout in the image sensor;

(3) FIGS. 2 and 3 propose two examples of hardware architecture of an HDR image generating system implementing the method illustrated in FIG. 1.

5. DETAILED DESCRIPTION OF THE INVENTION

(4) The general principle of the invention is based on the reconstruction of high dynamic range (HDR) images from images obtained by non-destructive readouts of an image sensor, according to an adaptive method, making it possible to substantially improve the acquisition speed of the HDR image, and to dynamically improve the number of non-destructive readouts necessary, based on the dynamics of the targeted HDR image, and brightness parameters of the scene to be imaged.

5.1 Notations, Definitions and Principles

(5) In the remainder of this document, the photoelectric conversion elements underlying the pixels of the sensor of the image acquisition system are made, for example, based on CMOS (Complementary Metal Oxide Semiconductor) technology. It will be recalled that CMOS technology equips most photographic or video systems. Image detectors of the CMOS type have the advantage of being adapted to be read in a so-called non-destructive readout (NDRO) mode.

(6) The non-destructive readout mode makes it possible to read the electric charges accumulated by each of the photoelectric conversion elements of the sensor (therefore the signal values associated with the pixels), without resetting the latter. In other words, the NDRO readout makes it possible to observe the formation of an image on the detector during an exposure without destroying it.

(7) Saturation refers to a state in which the quantity of incident light on the photoelectric conversion elements of the sensor exceeds the maximum quantity, in the linear operating range of the detector, of electric charges that these conversion elements can accumulate. This results in an overexposure phenomenon of the corresponding areas of the image, if one limits oneself to the linear range of the sensors. However, some sensors can use two responses, linear and logarithmic, at the same time, making it possible to lift this constraint, in particular in the choice of the accumulation time of the first NDRO.

(8) Below, we propose several notations and definitions that will be used in the remainder of this document. ADU: refers to an analog-digital unit, corresponding to the quantity of analog signal at the input of an analog-digital converter, so that it delivers a unit as output. Thus, by knowing the system gain (expressed in e-/ADU (electrons per ADU) for example), it is possible to determine the quantity of electrons from the detector, and by knowing the quantum efficiency, it is possible to determine the quantity of incident photons. Q.sub.: quantum efficiency of the system, defined as the ratio between the number of photons, at a given wavelength , having fallen on the sensor relative to the number of electric charges created by the photoelectric conversion elements of the sensor and usable as signal by the electronic readout system of the image acquisition system. The electric charges created by the photoelectric conversion elements of the detector, or sensor, from incident photons but next lost in the overall readout noise are therefore deduced here. F.sub.max,: maximum incident luminous flux at a given wavelength , which can be expressed for example in photons per second. F.sub.min,: minimum incident luminous flux at a given wavelength , which can be expressed for example in photons per second. T.sub.timeout: Maximum exposure time of the sensor for an image, expressed in seconds. This maximum exposure time is determined before any image acquisition time, and can be adjustable for each new HDR image generation phase. n.sub.timeout: index of the n.sup.th NDRO image, associated with an exposure time equal to the maximum exposure time T.sub.timeout. T.sub.P: exposure time for an HDR image to be generated, expressed in seconds, during which the non-destructive readouts of the sensor are done. The exposure time T.sub.P is less than or equal to the maximum exposure time T.sub.timeout and corresponds to the equivalent exposure time of the last NDRO image used for the reconstruction of the HDR image. t.sub.min: minimum accumulation time of the image acquisition system, corresponding to the minimum accumulation time until the first possible non-destructive readout of the sensor, expressed in seconds. t.sub.accu1: accumulation time until the first NDRO image useful for the HDR reconstruction. This time t.sub.accu1 corresponds to the exposure time associated with the first of the successive NDRO images used for the reconstruction of the HDR image. t.sub.NDRO: readout time, expressed in seconds, of an NDRO image. S.sub.min: weakest signal on the final HDR image generated according to an embodiment of the present invention. s.sub.min: weakest signal due to the weakest incident flux on the different successive NDRO images obtained by non-destructive readout of the sensor of the image acquisition system. S.sub.max: strongest signal on the final HDR image generated according to an embodiment of the present invention. s.sub.max: strongest signal due to the strongest incident flux on the different successive NDRO images obtained by non-destructive readout of the sensor of the image acquisition system. S.sub.sat: saturation signal of the sensor (imposed by the first saturated element in the detector: pixel, charge conversion and/or amplification chain, etc.) S.sub.0: signal corresponding to the weakest usable electronic signal excluding noise coming from the detector. This is typically the signal whose level corresponds to the readout noise of the detector (or more generally of the imaging system). S.sub.sel: signal selection threshold for selecting the value of the pixels for the HDR reconstruction according to a second embodiment of the invention described hereinafter. N.sub.r: readout noise of the detector, or more generally of the imaging system. N.sub.T: thermal noise associated with the thermal signal of the detector (accumulated during the exposure). N.sub.p: Photon noise associated with the light signal accumulated during the exposure. The noises N.sub.r, N.sub.T and N.sub.p are generally expressed in electrons brought to the output of the detector, or in ADU with knowledge of the system Gain in e-/ADU (electrons per ADU). P.sub.i,j.sup.NDRO.sup.n: Value of the signal associated with the pixel with coordinates (i,j) in the matrix of the sensor, coming from the NDRO image with n rank NDRO.sub.n, where 0nn.sub.timeout. RSB.sub.i,j.sup.NDRO.sup.n: Signal-to-Noise Ratio of the pixel with coordinates (i,j) coming from the NDRO image with n rank NDRO.sub.n, where 0nn.sub.timeout.

(9) It will be noted that the different signals S.sub.max, S.sub.max, S.sub.min, S.sub.min, S.sub.sat, S.sub.sel, S.sub.0 are described above with no associated unit. Indeed, if they are seen after analog-digital conversion, they will be expressed in ADU units. Conversely, if they are considered before the analog-digital conversion step, they will then be expressed in the unit of the physical property (analog) that characterizes them: Volts or Amperes.

(10) Based on these notations, the dynamic D of the HDR image to be generated can be expressed as the ratio:

(11) $\begin{array}{cc}D=\frac{S_{\max }}{S_{\min }}=\frac{s_{\max }/t_{\mathrm{accu}1}}{s_{\min }/T_p}& \left(\mathrm{Eq}1\right)\end{array}$
with
s.sub.min=T.sub.pF.sub.min,Q.sub.(Eq2)
S.sub.max=t.sub.accu1F.sub.max,Q.sub.(Eq3)
One can therefore deduce the relationship:

(12) $\begin{array}{cc}D=\frac{F_{\max }}{F_{\min }}& \left(\mathrm{Eq}4\right)\end{array}$
and:
t.sub.accu1=t.sub.min+(n.sub.1t.sub.NDRO)n.sub.10(Eq5)
T.sub.p=t.sub.min((n.sub.1+n.sub.2)t.sub.NDRO)n.sub.2n.sub.1(Eq6)
where the index n.sub.1 designates the index, from among a series of NDRO images obtained by non-destructive readout of the image sensor, of the first NDRO image to be used to generate the high dynamic range image HDR, and where the index n.sub.2 designates the number of NDRO images taken from the first useful image with index n.sub.1, such that n.sub.1+n.sub.2 is the index, from among the series of NDRO images obtained by non-destructive readout of the image sensor, of the last NDRO image to be used to generate the high dynamic range image HDR.

(13) The dynamic of the image acquisition system (with the detector in the first instance) is finite, it is limited at the low values by the highest noise level (S.sub.0) and at the high values by the saturation level of the system S.sub.sat (the first element of the chain that saturates). The incident luminous fluxes that can be rendered on a raw (non-HDR) image are those which, integrated during an exposure time t, yield a signal Sig that responds to S.sub.0<Sig<S.sub.sat. The luminous fluxes being set by the scene to be imaged and the diaphragm of the optical path that are, a priori, not changeable, the adjustable parameters are therefore the different exposure times with, in particular, T.sub.P, which corresponds to the exposure time associated with the last NDRO image used to generate the HDR image, and therefore the exposure time of this HDR image, and t.sub.accu1, which, as indicated above, corresponds to the exposure time associated with the first NDRO image used to generate the HDR image.

(14) These two adjustable parameters are, however, adjustable within certain limits, namely: t.sub.accu1t.sub.min: indeed, the exposure time associated with the first NDRO image to be used to generate the HDR image cannot be less than the minimum accumulation time of the image acquisition system; and
T.sub.T.sub.timeout(Eq7) Indeed, the exposure time associated with the HDR image is bounded by the maximum exposure time defined beforehand.

(15) It will be noted that the times t.sub.min and T.sub.timeout are fixed for a sequence of HDR shots, but can be modified to be better suited to the brightness conditions of the scene to be imaged. Nevertheless, T.sub.timeout must be short enough for the noise associated with the thermal signal accumulated during T.sub.timeout not to be greater than the minimum signal to be detected. The temperature, during operation, of the detector being known, the thermal signal (in electrons/second created, accumulated then read) is known, as is its associated noise. The thermal signal (also called thermal noise) as a function of the temperature, is a builder datum of the detector.

(16) Furthermore, the inventors of the present patent application have established that certain additional constraints should be respected by the minimum accumulation time of the system t.sub.min, in order to avoid any loss of information in a linear response system, so as not to depart from the linear operating range of the sensor.

(17) Thus, in the limit case where t.sub.accu1=t.sub.min, and where the accumulation time associated with the n.sup.th NDRO image is t.sub.accu_n=t.sub.accu1 ((n1)t.sub.NDRO), i.e., the accumulation times associated with the different successive NDRO images intersect by a pitch t.sub.NDRO, an additional constraint must be respected between the accumulation time of the first and the second useful NDRO images, in the form:

(18) $\begin{array}{cc}\frac{t_{\mathrm{accu\_}2}}{t_{\mathrm{accu}1}}<\frac{S_{\mathrm{sat}}}{S_0}\mathrm{or}& \\ \frac{t_{\min }+\left(t_{\mathrm{NDRO}}\right)}{t_{\min }}<\frac{S_{\mathrm{sat}}}{S_0}& \left(\mathrm{Eq}.8\right)\end{array}$

(19) In other words, to avoid any loss of information, it is appropriate, for systems with linear response sensors, to meet the conditions of equation Eq. 8. Alternatively, it is possible to use systems with linear/logarithmic mixed response sensors, which lift the issue of relative accumulation time between the first and second useful NDRO images, t.sub.accu1 and t.sub.accu_2, which makes it possible to reduce the minimum accumulation time of the acquisition system t.sub.min to the minimum physical time of the system clock (i.e., to one clock tick).

5.2 Description of One Embodiment

(20) These principle having been recalled, we will now describe, in connection with FIG. 1, a flowchart of a method for generating a high dynamic range image HDR, according to one embodiment of the present invention.

(21) Such a method can be implemented in any type of image acquisition system, whether it involves still or video images. Such a system is in particular described in more detail hereinafter in connection with FIGS. 2 and 3.

(22) According to this embodiment, a real-time selection is done of the values of the pixels to be used for the reconstruction of the final HDR image with only the storage of the single image at any moment of the generation.

(23) Indeed, from the first read NDRO image (readout done after the accumulation time t.sub.min), it is necessary to determine the pixels, coming from this first NDRO image and the NDRO images that follow, that will be relevant for the HDR reconstruction. Indeed, a value of a pixel coming from the n.sup.th NDRO image, P.sub.i,j.sup.NDRO.sup.n, will only be usable if at least

(24) $S_0{<}^{P_{i,j}^{{\mathrm{NDRO}}_n}}<S_{\mathrm{sat}}$
where S.sub.0 is the signal corresponding to the weakest usable electronic signal coming from the detector (typically, the signal whose level corresponds to the readout noise of the detector or more generally of the imaging system), and where S.sub.sat is the saturation signal the sensor (imposed by the first saturated element in the detector: pixel, charge conversion and/or amplification chain, etc.). In this embodiment, one seeks to optimize the SNR (Signal-to-Noise Ratio) on a level set in advance in the shortest time, not exceeding a maximum exposure time (T.sub.timeout) that is also set in advance.

(25) $\begin{array}{cc}{\mathrm{RSB}}_{i,j}^{{\mathrm{NDRO}}_n}=\frac{P_{i,j}^{{\mathrm{NDRO}}_n}}{\sqrt{N_r^2+N_T^2+N_p^2}}& \left(\mathrm{Eq}.9\right)\\ \mathrm{with}& \\ N_{\mathrm{tot}}=\sqrt{N_r^2+N_T^2+N_p^2}\mathrm{or}{\mathrm{RSB}}_{i,j}^{{\mathrm{NDRO}}_n}=\frac{p_{i,j}^{{\mathrm{NDRO}}_n}}{N_{\mathrm{tot}}}& \left(\mathrm{Eq}.10\right)\end{array}$

(26) The value of the noise associated with the value of a pixel evolves at the root of the value of the signal accumulated on the pixel. To maximize the signal-to-noise ratio RSB.sub.i,j of a pixel with coordinates (i,j), it is necessary to accumulate as much signal as possible to increase its value P.sub.i,j, i.e., to maximize its exposure time: indeed, P.sub.i,j evolves as a function of exposure time and the noise also evolves, at the root of P.sub.i,j, therefore much less quickly. In most cases, this function is linear (with a constant incident luminous flux during exposure). This allows the direct use of the NDRO images by simple weighting (Eq. 13, see below). In the case of a nonlinear behavior, it is necessary to obtain the response function of the detector (and/or of the imaging system) for each pixel, during the calibration of the detector and/or the associated imaging system. This response for each pixel will make it possible to weight the value of each pixel, and thus to fall back on the linear exploitation described in the equation (Eq. 13). The SNR is optimized at each NDRO and pixels are selected on this criterion. Thus, the final SNR will be optimized for each selected pixel in each respective NDRO and in particular for the weakest signals that have been selected and will have led to the last NDRO (as long as the desired SNR has been reached for the pixels with the weakest signals before the timeout t.sub.out, also set in advance).

(27) As illustrated in FIG. 1, during a prior step referenced 100, one determines the value of certain adjustable parameters of the system, which include: the minimum accumulation time of the acquisition system t.sub.min; the maximum exposure time of the sensor T.sub.timeout; the desired signal-to-noise ratio SNR for each of the pixels and above all for the pixels with the weakest signals, for the HDR image to be generated (see Eq. 10); the value of a single selection threshold S.sub.sel, which is defined such that S.sub.sel<S.sub.sat (and S.sub.sel>S.sub.0).

(28) The value of these parameters is in particular established based on brightness conditions (luminous flux) of the scene to be imaged. Thus, the minimum accumulation time of the acquisition system t.sub.min, which is set before the shot acquisition, can be optimized, before the acquisition, at the strong luminous flux of the scene. For an acquisition system with a linear response sensor, this minimum accumulation time t.sub.min must, however, respect the constraint of equation Eq. 8:

(29) $\frac{t_{\min }+\left(t_{\mathrm{NDRO}}\right)}{t_{\min }}<\frac{S_{\mathrm{sat}}}{S_0},$
in order to stay within the linear operating range of the sensor, and to avoid any loss of information. For an acquisition system with a linear/logarithmic mixed response sensor, it is possible to reduce the time t.sub.min if applicable to the minimum time of the system clock.

(30) Likewise, the signal selection threshold S.sub.sel can be modified to best adapt to the illumination conditions of the scene, or to optimize and favor the exposure time T.sub.P, or to optimize and favor the final SNR for the weakest signals, etc.

(31) This prior step referenced 100 must be implemented upon initialization of the image acquisition system. It can be reiterated upon each new image acquisition for HDR reconstruction. Alternatively, the same parameters can be kept for several successive HDR image captures. It is also possible to consider that some of these parameters remain unchanged from one HDR shot acquisition to the next, while others are adapted upon each new shot acquisition.

(32) During the step referenced 101, one begins a series of non-destructive readouts of the sensor, which each deliver a so-called NDRO image with index n, n being initialized at 0 (step referenced 102).

(33) For each current NDRO image with index n, the value of the pixel P.sub.i,j.sup.NDRO.sup.n is selected to be used for the reconstruction of the final HDR image once:
P.sub.i,j.sup.NDRO.sup.nS.sub.sel(Eq. 11)

(34) This comparison of the value of the pixels P.sub.i,j.sup.NDRO.sup.n of the current image NDRO.sub.n with the signal selection threshold S.sub.sel is done during a step referenced 103. When the condition is met, the selected pixel is stored (step referenced 104), in association with the exposure time of the image NDRO.sub.n from which it is extracted, denoted t.sub.i,j.sup.NDROn. It will then be used to generate the HDR image referenced 108.

(35) The higher S.sub.sel is (while verifying S.sub.sel<S.sub.sat), the more pixels P.sub.i,j.sup.NDRO.sup.n there are whose value meets the condition of equation Eq. 11, in a time shorter than T.sub.timeout, with a high SNR.

(36) The first NDRO image in which at least one pixel satisfies the condition of equation Eq. 11 is the first image from the series of NDRO images that will be used for the reconstruction of the HDR image, referenced 108.

(37) If no pixel of the current image NDRO.sub.n satisfies the condition of equation Eq. 11, one verifies, during a step referenced 105, whether the exposure time t.sub.i,j.sup.NDROn associated with the current image NDRO.sub.n reaches the maximum exposure time determined during the preliminary step 100.

(38) If this is not the case, the index of the current NDRO image is incremented during a step referenced 106 (n:=n+1), and the non-destructive readouts of the sensor continue.

(39) If, however, the result of the comparison of the step referenced 105 is positive, this means that for some pixels of the NDRO image associated with the maximum exposure time, P.sub.i,j.sup.NDRO.sup.n remains below S.sub.sel. In this case, the pixel, among the n.sub.2 NDRO images taken, which would yield the best SNR, will be that of the last NDRO image taken, associated with the predefined maximum exposure time.

(40) One then stores, during a step referenced 107, the values of the pixels of this last useful NDRO image, in association with the maximum exposure time that is associated with them, whose value remained below the selection threshold S.sub.sel.

(41) Indeed, while for at least some pixels of the sensor, the condition of equation Eq. 11 is not verified before the end of the exposure time T.sub.P, it is then the last pixel value coming from the last NDRO image taken that will be kept, in association with its exposure time (here corresponding to T.sub.P).

(42) The non-destructive readout iterations of the sensor can be interrupted when all of the pixels of the sensor have reached a signal value greater than or equal to the signal selection threshold S.sub.sel.

(43) Each pixel selected for the reconstruction of the HDR image will therefore come from one of the NDRO images taken between t.sub.min and T.sub.P with T.sub.PT.sub.timeout, it will be characterized by its value P.sub.i,j.sup.NDRO.sup.n and its exposure time
t.sub.i,j.sup.NDRO.sup.n=t.sub.min+((n1)t.sub.NDRO)(Eq. 12).

(44) The generation of the HDR image, referenced 108, is done from values of the pixels stored during steps referenced 104 and 107, weighted by their respective exposure times:

(45) $\begin{array}{cc}{P}_{i,j}^{\mathrm{HDR}}=\frac{P_{i,j}^{{\mathrm{NDRO}}_n}}{t_{i,j}^{{\mathrm{NDRO}}_n}}& \left(\mathrm{Eq}.13\right)\end{array}$

(46) It will be noted that, in the case where the acquisition time t.sub.min does not comply with the constraint of equation Eq. 8, all of the pixels of the final HDR reconstructed image, whereof the value

(47) $P_{i,j}^{\mathrm{HDR}}=\frac{P_{i,j}^{{\mathrm{NDRO}}_n}}{t_{i,j}^{{\mathrm{NDRO}}_n}}$
is such that

(48) 0$\frac{S_0}{t_{\min }}<P_{i,j}^{\mathrm{HDR}}<\frac{S_{\mathrm{sat}}}{t_{\mathrm{accu\_}2}},$
will not contain relevant information.

(49) In one embodiment, an image of the matrix of the detector is stored in a memory and is re-updated upon each new NDRO image taken. Each element of this memory area stores the value P.sub.i,j.sup.NDRO.sup.n (current during the last read NDRO image) weighted by its corresponding exposure time t.sub.i,j.sup.NDRO.sup.n. For the pixels whose value has reached the selection threshold S.sub.sel in a preceding NDRO image, the elements of the memory zone are of course not re-updated, and remain frozen at the value stored during the step referenced 104.

(50) Thus, during the first NDRO readout of the step referenced 101, the memory area representative of the detector matrix is initialized with the values of the pixels P.sub.i,j.sup.NDRO.sup.n of the first NDRO image with index n=0. Those of these pixels whose value is above the selection threshold of the signal S.sub.sel are stored permanently in the memory area, without being able to be updated further during the following NDRO readouts.

(51) The reconstruction of the high dynamic range image HDR is therefore done gradually, over the course of the different non-destructive readouts of the sensor done during the exposure time T.sub.P.

(52) This embodiment, illustrated in FIG. 1, allows the real-time adaptation of the exposure times (always with the prerequisite of equation Eq. 7). This embodiment does not require determining the NDRO images at the moments T.sub.p and t.sub.accu1, the HDR image building itself upon each new NDRO image, until the previously defined optimum. Furthermore, this method works optimally with one and only one image of the matrix of the detector stored in memory. This matrix, in memory, is re-updated upon each new acquired NDRO image. Each element of this matrix stores a value of the pixel, whether it is current during the last read NDRO image or of the n.sup.th NDRO, and is weighted by its respective exposure time.

(53) This method is therefore particularly advantageous in terms of required memory space, and necessary computing capacity to generate the HDR image. Unlike certain methods of the state of the art, it is not necessary to access several NDROs at once. As a result, during a real-time HDR generation, the number of NDROs is then not limited by the system.

(54) It will also be noted that, in the case of a system with a linear/logarithmic mixed response sensor, there is cause to have a calibration of the logarithmic response and the linear/logarithmic transition area, in order to weight the value of each pixel, and thus to fall back on the linear exploitation described by equation Eq. 13. Furthermore, also in this case, the selection threshold for the pixels must be adapted to the operating zone of the sensor. Thus, for the first two NDRO images, the selection threshold S.sub.sel can use the logarithmic part of the operation of the sensor, while being such that S.sub.sel>S.sub.sat_lin, where S.sub.sat_lin is the maximum signal level at which the linear/logarithmic detector offers a linear response. Being in a logarithmic response, there is no longer a relevant maximum saturation limit at this time. Beyond the second NDRO image, all of the following NDRO images being read after an accumulation time regularly increasing by a pitch t.sub.NDRO, one is in a linear operating range of the sensor. One then has S.sub.selS.sub.sat_lin, where the value S.sub.sat_lin becomes equivalent to the value S.sub.sat of the saturation signal of the sensor, mentioned previously in the present document.

5.3 Examples of HDR Image Generation Systems

(55) We will now describe, in connection with FIGS. 2 and 3, two examples of hardware architecture of an HDR image generation system (200, 300) implementing the method described above in connection with FIG. 1.

(56) The elements shared by FIGS. 2 and 3 bear the same numerical references, and are briefly described hereinafter.

(57) Such a system for generating HDR images 200, 300 comprises the following elements, which are connected to one another by a data bus and addresses: a processing and digital control unit 201, 301, which can for example be a microprocessor, an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit); a memory 202, 302; a user interface, not shown, connected to the processor 201, 301 by an external digital link 205, 305 of the Ethernet, USB, Cam Link, etc. type; a digital processing module 203, 303, in particular comprising an image sensor 204, 304 comprising a plurality of pixels arranged in matrix form, and each associated with a photoelectric conversion element making it possible to convert received light into electric charges and to accumulate these electric charges for an exposure time to the light. This sensor is adapted to operate in a non-destructive readout mode.

(58) Such a system 200, 300 also conventionally comprises a memory of the ROM (Read Only Memory) type that comprises at least one program and different parameters necessary for the execution of the method according to one embodiment of the invention. When it is powered on, the processor loads the program into a memory of the RAM (Random Access Memory) type and executes the corresponding instructions.

(59) The system 200, 300 also comprises an electrical power source, not shown, for example in battery form, which in particular provides the different power signals 214, 314.

(60) The user interface allows the user to choose the determined parameters during the step referenced 100 in FIG. 1. They supply a module 211, 311 (User external Controls & Values interpreter), in which they are stored, and which is adapted to provide them to the other elements of the system 200, 300. This module referenced 211, 311 in particular stores: the maximum exposure time of the sensor T.sub.timeout; the saturation signal S.sub.sat; the minimum signal S.sub.0.

(61) A processor 212, 312 is configured to automatically calculate the signal selection threshold S.sub.sel, from the saturation signal S.sub.sat, the minimum signal S.sub.0, and information coming from the preceding HDR image acquisition, such that S.sub.0<S.sub.sel<S.sub.sat.

(62) The maximum exposure time of the sensor T.sub.timeout is also sent to a clock generator 213, 313, intended to generate and control the different synchronous clock signals necessary for the operation of the analog processing module 203, 303.

(63) The memory 202, 302 comprises several memory cells (i,j), which each contain the value of the pixel P.sub.i,j.sup.NDRO.sup.n selected during the step referenced 104 or 107 in FIG. 1. Each memory cell also stores the index n of the current NDRO image from which this pixel value is extracted. The exposure time associated with the pixel is n*t.sub.NDRO.

(64) Upon each non-destructive readout of the sensor 204, 304, the latter delivers an analog signal, representing the value of the electric charges accumulated by each of the pixels of the sensor.

(65) As illustrated previously in connection with FIG. 1, the values of the pixels P.sub.i,j.sup.NDRO.sup.n are tested relative to the signal selection threshold S.sub.sel (Threshold Value) simply by level comparison (voltage level, for example).

(66) In the example of FIG. 2, this comparison is done, very simply, in an analog manner, by using comparators 215. After amplification 216, the pixel values conveyed by the analog signal coming from the sensor 604 are compared to the signal selection threshold S.sub.sel (Threshold Value) and the saturation signal S.sub.sat (Saturation Value). After application and analog-digital conversion 217, the pixels whose value is above the selection threshold S.sub.sel must be stored, and are therefore transmitted by a bus to the memory 202.

(67) Due to its simplicity, this analog preprocessing (for the adaptive aspect of this HDR reconstruction method by NDRO image) can be integrated directly, On Chip, within the sensor, for pixel clusters, or directly within each pixel in a 3D-CMOS structure, for example.

(68) In the example of FIG. 3, on the contrary, this comparison is done in an offloaded manner within the digital processing unit 301, in a comparator module 315. Thus, the analog signal delivered by the sensor 304 is amplified and converted into a digital signal (module 317), before being supplied at the input of the comparator module 315, which compares the digital signal value associated with each pixel to the signal selection threshold S.sub.sel and to the saturation signal S.sub.sat, and selects those which must be stored in the memory 302, so that they may be used to reconstruct the HDR image.