Method for designing a passive single-channel imager capable of estimating depth of field

Abstract

A computer-implemented method for designing an electro-optical imaging system for estimating the distance of a source includes use of an optical subsystem, a detector subsystem and a digital image processing subsystem. The method includes the modelling of the propagation of radiation from its source through the optical subsystem, the detector subsystem and the digital image processing subsystem; the modelling being based on a spatial model of the source; the method including a joint step of simultaneously designing the optical subsystem and the digital image processing subsystem, the designing step being based at least on one performance metric depending on a comparison between the local estimation of the distance from the source and the actual distance from the source.

Claims

1. A computer-implemented method for designing an electro-optical imaging system for estimating a distance between a point of arrival of the electro-optical imaging system and a source, the electro-optical imaging system including an optical subsystem, a detector subsystem and a digital image processing subsystem, the method comprising: modeling propagation of radiation from the source through the optical subsystem, the detector subsystem, and the digital image processing subsystem, the modeling based, at least in part, on a spatial model of the source; simultaneously determining design parameters for the optical subsystem and the digital image processing subsystem based, at least in part, on a performance metric, wherein said performance metric is determined using a comparison function that compares an estimate of the distance between the point of arrival and the source and the actual distance between the point of arrival and the source; and providing an electro-optical imaging system including the optical subsystem, the detector subsystem, and the digital image processing subsystem conforming to the determined design parameters for the optical subsystem and the digital image processing subsystem; wherein said comparison function comprises calculating a probabilistic average of the behavior of the discrepancy between the estimated distance and the actual distance for a catalog of images; and wherein said discrepancy is characterized by calculating an average curvature of a likelihood function, as regards the depth, for the catalog of images, such likelihood function being calculated according to the following acts: recording a library of point-spreading functions (PSFs), with each one of said PSF functions being associated with a depth and with one of a plurality of different electro-optical systems; at least one of capturing and modeling at least one image using one of the PSFs of the PSF library; dividing said at least one image into at least one thumbnail having a preset size to record a thumbnail collection; calculating a likelihood of each PSF, as regards the content of the thumbnails, and estimating the point-spreading function (PSF), which maximizes this likelihood for each one of said thumbnails; and measuring a local curvature of the likelihood function around the PSF to maximize the likelihood function.

2. The method according to claim 1, wherein said likelihood function is calculated from the spatial model of the source.

3. The method according to claim 2, wherein said model is at least one of: a power spectral density; a covariance matrix learned from an image database, and; a precision matrix, based on an isotropic and Gaussian distribution of source gradients.

4. The method according to claim 1, further comprising simultaneously acquiring and processing a plurality of images with different focuses.

5. The method according to claim 4, wherein said plurality of images is obtained with a detector consisting of a plurality of arrays of sensors, with each array of the plurality of arrays of sensors being sensitive to a specific spectral band, and with each image of the plurality of images corresponding to one of said spectral bands.

6. The method according to claim 5, further comprising adjusting a focus for each one of said spectral bands with the optical subsystem.

7. The method according to claim 1, further comprising varying an encoding of an encoded diaphragm, and determining a resulting discrepancy resulting therefrom.

8. The method according to claim 1, further comprising optimizing the determined parameters for the optical subsystem and the digital image processing subsystem in real time to facilitate estimation of the distance between the point of arrival and the source with conditions of use, the conditions at least one of provided by a user, and estimated by algorithms corresponding to the electro-optical system.

9. The method according to claim 1, wherein determining design parameters comprises combining the performance metric with another metric characterizing an ability to produce an image having a large depth of field.

10. The method according to claim 1, wherein the performance metric is compared to a template set by a user.

11. The method according to claim 10, wherein if several hypothetical electro-optical systems match the template, the several hypothetical electro-optical systems are proposed to the user, and options to impose more constraints are provided to the user.

12. The method according to claim 10, further comprising informing the user if no hypothetical electro-optical system matches the template, and prompting the user to reduce the constraints.

13. An electro-optical device comprising: a depth estimator operating in a distance range; an objective comprising a number of lenses; a detector positioned in a focal plane of the objective; and a diaphragm; wherein characteristics of said objective are determined according to a performance metric depending on a comparison function that performs a comparison between a local estimate of a distance from a source to a point of arrival of the electro-optical device and an actual distance from the source to the point of arrival; wherein said comparison function comprises calculating a probabilistic average of the behavior of the discrepancy between the estimated distance and the actual distance for a catalog of images; and wherein said discrepancy is characterized by calculating an average curvature of a likelihood function, as regards the depth, for the catalog of images, such likelihood function being calculated according to the following acts: recording a library of point-spreading functions (PSFs), with each one of said PSF functions being associated with a depth and with one of a plurality of different electro-optical systems; at least one of capturing and modeling at least one image using one of the PSFs of the PSF library; dividing said at least one image into at least one thumbnail having a preset size to record a thumbnail collection; calculating a likelihood of each PSF, as regards the content of the thumbnails, and estimating the point-spreading function (PSF), which maximizes this likelihood for each one of said thumbnails; and measuring a local curvature of the likelihood function around the PSF to maximize the likelihood function.

14. An electro-optical device according to claim 13, wherein at least one lens of said number of lenses is of conical type.

15. A method of manufacturing an electro-optical device, the method comprising: modeling, with a computer, propagation of radiation from a source through a plurality of different hypothetical optical subsystems for the electro-optical device using a spatial model of the source, wherein each hypothetical optical subsystem of the plurality of different hypothetical optical subsystems is associated with a different set of optical parameters; modeling detection of the radiation for a hypothetical detector subsystem of the electro-optical device for each hypothetical optical subsystem of the plurality of different hypothetical optical subsystems; modeling, for each hypothetical optical subsystem, image processing of data corresponding to the radiation detected by the detector subsystem for a plurality of different hypothetical image processing subsystems for the electro-optical device, wherein each hypothetical image processing subsystem is associated with a different set of processing parameters; computing a performance metric of the electro-optical device for each combination of the plurality of different hypothetical optical subsystems and the plurality of different hypothetical image processing subsystems, wherein the performance matrix is determined using a comparison function that compares an actual distance between a point of arrival of the electro-optical device and the source to an estimate of the actual distance, wherein said comparison function comprises calculating a probabilistic average of the behavior of the discrepancy between the estimated distance and the actual distance for a catalog of images, said discrepancy being characterized by calculating an average curvature of a likelihood function, as regards the depth, for the catalog of images, such likelihood function being calculated according to the following acts: recording a library of point-spreading functions (PSFs), with each one of said PSF functions being associated with a depth and with one of a plurality of different electro-optical systems; at least one of capturing and modeling at least one image using one of the PSFs of the PSF library; dividing said at least one image into at least one thumbnail having a preset size to record a thumbnail collection; calculating a likelihood of each PSF, as regards the content of the thumbnails, and estimating the point-spreading function (PSF), which maximizes this likelihood for each one of said thumbnails; and measuring a local curvature of the likelihood function around the PSF to maximize the likelihood function; selecting a combination of one of the plurality of different hypothetical optical subsystems and one of the plurality of different hypothetical image processing subsystems based, at least in part, on the performance metric corresponding thereto; and manufacturing the electro-optical device including: an optical subsystem having a same set of optical parameters as the selected one of the plurality of different hypothetical optical subsystems; and an image processing subsystem having a same set of processing parameters as the selected one of the plurality of different hypothetical image processing subsystems.

16. The method of claim 15, wherein the set of optical parameters comprises at least one of a focal length of lenses of the optical subsystem, a number of lenses of the optical subsystem, radii of curvature of the lenses, diameter of the lenses, conicity of the lenses, and materials of the lenses.

17. The method of claim 15, wherein the set of processing parameters includes at least one of point spreading function parameters, and a size of image thumbnails.

18. The method of claim 15, wherein the comparison function comparing the actual distance to the estimated distance compares a distance between a center point of an entrance pupil of the optical subsystem and the source to an estimate of the distance between the center point of the entrance pupil and the source.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The present invention will be better understood upon reading the detailed description of an exemplary non-restrictive embodiment that follows, while referring to the accompanying drawings, wherein:

(2) FIG. 1 shows the schematic diagram of the various functional blocks of an electro-optical passive single-channel system estimating the depth according to the state of the art;

(3) FIG. 2 shows the schematic diagram of the design-assisted method provided;

(4) FIG. 3 shows the curve of the theoretical estimation accuracy obtained when using a calculation of depth estimation performance for two optical systems having the same focal length and the same opening but different diaphragm shapes;

(5) FIG. 4 shows the curve of the theoretical estimation accuracy obtained when using a calculation of depth estimation performance for two different focuses in the same optical system;

(6) FIG. 5 shows the curve of theoretical estimation accuracy obtained when using a calculation of depth estimation performance for two chromatic optical systems having different positions of the focus planes of the red, green and blue channels;

(7) FIG. 6 shows the lenses constituting an electro-optical system optimized to facilitate the estimation of depth;

(8) FIG. 7 shows a table of the characteristics of the lenses of the electro-optical system shown in FIG. 6; and

(9) FIG. 8 shows the curve of the theoretical estimation accuracy obtained when using a calculation of depth estimation performance for the electro-optical system shown in FIGS. 6 and 7.

DETAILED DESCRIPTION

(10) FIG. 1 shows the schematic diagram of the various functional blocks of an electro-optical passive single-channel system estimating the depth according to the state of the art.

(11) It knowingly comprises a detector, for example, a CCD detector associated with an optical subsystem, and a signal processing subsystem.

(12) The optical subsystem is characterized by parameters noted o, which typically correspond to: the focal length; the opening; the characteristics of the lenses constituting the optical system (number, radius of curvature, diameter, conicity, glass type); and the field.

(13) The detector is characterized by the d parameters, which typically correspond to the size and the number of pixels, or to other characteristics such as exposure time, or, for example, the possible presence of a Bayer filter. The optics/detector assembly forms the image of the scene, which will be used for the estimation of depth.

(14) The processing part of the electro-optical system uses the parameters of the point and scene spreading function, as well as processing parameters noted t, which correspond, for instance, to the size of the thumbnails of the image.

(15) The parameters of the source model, which are represented by x, are involved in the depth estimation. The hypothesis on the source corresponds, for example, to a local model of the contrasts that the electro-optical system can observe (textures, contours, etc.).

(16) Parameters Involved in the Precision of the Depth Estimation

(17) The precision of the depth estimation first depends on the variability of the Point Spreading Function (PSF) with depth. The latter depends on the optical detectors o and d parameters. Then, the precision depends on the parameter t, more particularly characterizing the number of pixels of each thumbnail used to estimate the depth.

(18) In addition, some parameters affecting the depth estimation depend on the conditions of use of the electro-optical system. These are, on the one hand, the noise level, which affects the images, which is broadly linked to the lighting conditions wherein the device is used. This noise level is defined here by the Signal-to-Noise Ratio (SNR). On the other hand, characteristics of the scene observed by the device are described by the parameters of the source model, x.

(19) The precision of the depth estimation of an electro-optical system thus depends both on the optical and sensor parameters (d and c), the processing parameters (t) and on the conditions of use and the model of the source (SNR, x).

(20) Description of an Exemplary Method for Designing an Electro-Optical Single-Channel System

(21) Designing an electro-optical system estimating the depth requires the joint optimization of all the parameters that affect the accuracy of the estimation according to the user's requirements as regards the precision template and the conditions of use.

(22) Let a user define, for example, the field of the electro-optical system, the SNR, the maximum overall dimensions of the electro-optical system and the desired precision over a depth range.

(23) An exemplary designing method according to the proposed invention consists in: taking an initial electro-optical system for which the conditions of use are close to those imposed by the user in a library of electro-optical systems; modelling a set of PSFs relative to such system, either using simple mathematical functions, or using wave optics formulas or using optical design software; calculating the precision metric of the depth estimation; comparing the result with the template imposed by the user; executing a feedback loop introducing successive modifications in the optical, detector and processing parameters with a comparison of the metric with the template for each modification; and selecting the electro-optical system that is the closest to the template to return the optimized parameters to the user.

(24) Advantageously, if several electro-optical systems meet the conditions imposed by the user, the latter can be offered several solutions or the possibility of imparting more constraints so as to re-optimize the system.

(25) On the contrary, if the imparted conditions are too strict and do not make it possible to optimize an electro-optical system, the user will be informed thereof in order to revise his/her constraints.

(26) The method described above is shown in FIG. 2.

(27) Evaluation of the Estimation Precision

(28) Advantageously, the precision metric of the depth estimation is calculated by a probabilistic average of the behavior of the discrepancy between the estimated distance and the actual distance in a catalog of images.

(29) Conventionally, such discrepancy is characterized by calculating the average curvature of the likelihood function, as regards depth, for a catalog of images. Such likelihood function is calculated according to the following steps: recording a library of point spreading functions (PSFs), with each one of said PSF functions being associated with a depth and an electro-optical system; capturing at least one real image, or modelling at least one image using one of the PSFs in the PSF library; dividing the thumbnail image(s) into thumbnail(s) having a predetermined size to record a thumbnail collection; calculating the likelihood of each PSF, as regards the thumbnails' content and estimating, for each one of said thumbnails, the PSF that maximizes this likelihood; and measuring the local curvature of the likelihood criterion around the PSF that maximizes the likelihood function.

(30) The measurement of this curve provides a value representative of the discrepancy between the estimated distance and the actual distance, referred to as the Cramer-Rao bound (CRB).

(31) According to an advantageous alternative solution, the precision of the depth estimation is calculated without it being necessary to acquire or to simulate a catalog of images, thanks to the use of a spatial model of the source.

(32) According to one particular alternative embodiment, the spatial model of the source is a power spectral density. Such modelling is particularly used in the patent EP1734746 mentioned in the state of the art.

(33) According to one particular alternative embodiment, the source is modelled using the covariance matrix learned from an image database.

(34) According to one particular alternative embodiment, the source is modelled using the precision matrix corresponding to the inverse of the covariance matrix, based on an assumption of isotropic and Gaussian function of the gradients of the source. Letting Qx be the precision matrix is an advantageous alternative solution and consists in writing:

(35) $\begin{array}{cc}\frac{D^{T}D}{{}_x^2}& \left(1\right)\end{array}$

(36) where D is the concatenation of a horizontal and vertical first order derivative operator, and .sub..sup.2 is a parameter characterizing the variance of the gradients of the source.

(37) According to a particular calculation, if Hp is the convolution matrix associated with the PSF of the depth p, modelling the source using the precision matrix makes it possible to write the CRB as follows:

(38) $\begin{array}{cc}\mathrm{BCR}\left(p\right)={\left[\mathrm{tr}\left(Q_p^{-1}\frac{d{Q}_p}{dp}{Q}_p^{-1}\frac{d{Q}_p}{dp}\right)\right]}^{-1}& \left(2\right)\end{array}$

(39) The quantity characterizing the depth estimation precision is then .sub.CRB(p), which is the square root of CRB(p).
Q.sub.P=R.sub.b.sup.1(IH.sub.p(H.sub.p.sup.TR.sub.b.sup.1H.sub.p+Q.sub.x).sup.1H.sub.p.sup.TR.sub.b.sup.1(3)

(40) where Rb is the noise covariance matrix and Qp is the data precision matrix.

(41) In one particular case, noise can be modelled as a random probability density white process and a Gaussian function with a variance .sup.2b and the data precision matrix can then be written:

(42) $\begin{array}{cc}=\frac{I-B_P}{{}_b^2}\mathrm{with}& \left(4\right)\\ B_P={H_P\left(H_P^T{H}_P+D^{T}D\right)}^{-1}{H}_P^T\mathrm{and}& \left(5\right)\\ =\frac{{}_b^2}{{}_x^2},& \left(6\right)\end{array}$

(43) which is interpreted as the reverse of the Signal-to-Noise Ratio (SNR).

Examples

(44) 1) Modifications in the Diaphragm to Help Estimate the Depth

(45) The point-spreading functions of an ideal optical system (i.e., with no aberration), are simulated using wave optics formulas. Such system has a focal length of 50 mm and an opening of 2, with pixels of 7 m. The focus plane is positioned at 1.9 m from the electro-optical system. The theoretical estimation precision obtained in the case of a conventional diaphragm is compared to the one obtained in the case of an encoded diaphragm, such as the one proposed in the article by A. Levin cited in the state of the art, for the same signal-to-noise ratio.

(46) FIG. 3 illustrates the gain of the encoded diaphragm with respect to a conventional diaphragm for the depth estimation.

(47) Such curves make it possible to compare various forms of diaphragm to find the one that facilitates the depth estimation.

(48) 2) Modification in the Focus to Help the Depth Estimation

(49) The point spreading functions of an ideal electro-optical system, i.e., which is free of aberrations, are simulated using wave optics formulas. Such system has a focal length of 20 mm and an opening of 3, with pixels of 7 m.

(50) The focus plane is positioned 1.4 m (green curve) and 2 m (blue curve) away from the device. The theoretical estimation precision of CRB obtained for the two focuses is compared.

(51) FIG. 4 shows that two different focuses do not have the same estimation precision curves.

(52) Each one more particularly has a peak of inaccuracy as regards the focus plane. Thus, to estimate a depth between 0.5 and 1 m, a 1.3 m focus should rather be used. Similarly, to estimate distances beyond 2.5 m, the 2m focus is more favorable. Such curves make it possible to adjust the focus of the device according to the region of space wherein depth is to be estimated.

(53) 3) Modifications in the Position of the Focus Planes of the Three Red, Blue, and Green Channels in the Case of Chromatic Optics

(54) FIG. 5 shows the theoretical depth estimation performances of two electro-optical chromatic systems having an opening of 3, a focal length of 25 mm, having a detector of the tri-CCD type, with pixels of 5 m. The focus and the axial chromatic aberration of both systems are set so as to have the focus planes of the green and blue channels at 1.8 m and 2.6 m, respectively, for both systems. The focus plane of the red channel is 4 m away for the first system and is ad infinitum for the second system. The CRB curves show that the most favorable system in the 1 to 5 m region is the system with a 4 m focus of the red channel.

(55) 4) Optimization of a Chromatic Electro-Optical System

(56) FIGS. 6 and 7 show the characteristics of an optimized chromatic electro-optical system using the method illustrated in FIG. 2. FIG. 8 shows the CRB curves of such imager over a depth range.

(57) Detailed Description of the Electro-Optical Device

(58) The present example relates to an exemplary optical device comprising means for estimating the depth.

(59) The device illustrated in FIG. 6 comprises an objective formed, in the example described by way of non-restrictive example, of four lenses 10 to 13 and a detector 14 positioned in the focal plane of the objective.

(60) The lens 13 is of the conical type to correct certain optical aberrations.

(61) A diaphragm 15 is positioned in the optical path of the objective. In the example described, this diaphragm is of a conventional type, but it may also be formed by an encoded diaphragm.

(62) The invention relates to the optimization of this known objective to maximize the precision of the depth estimation, within a determined range [P.sub.min, P.sub.max] shown in FIG. 8.

(63) Such optimization may be executed either experimentally and empirically or by simulation.

(64) The empirical method consists in generating a plurality of objectives, with lenses having different index, positioning and curvatures characteristics, with the following constraints: constant number N of lens; constant focal length of the objective for a reference wavelength; constant opening of the objective lens; unchanged detector; and the signal-to-noise ratio of the observed light source.

(65) The variable parameters are: the radius of curvature of each one of the N lenses; the nature of the material (and hence index) of each one of the N lenses; the relative position of each one of the N lenses; the conicity of the conical lens 12 when such lenses are provided; the position of the detector 14; and the position and the configuration of the diaphragm 15.

(66) For each one of the optical systems, a point light source having a wide spectral band is positioned at a distance p between P.sub.min and P.sub.max, preferably on the optical axis, and at least one chromatic image produced with the M-channel detector 14 is acquired, with such image being referred to as PSF.sub.m(p) for m=1, . . . , M. For example, the detector comprises three RGB channels. In the described example, M is equal to three and m=R, G, B is acquired for a chromatic image triplet PSF.sub.m(p). Each image in the triplet makes it possible to calculate a convolution matrix H.sub.m Using the M matrices H.sub.m, a chromatic matrix H.sub.p is then obtained with the following equation:

(67) $H_P\left(p\right)=\left(\begin{array}{ccc}{c}_{1,1}{H}_R& c_{1,2}{H}_R& c_{1,3}{H}_R\\ c_{2,1}{H}_G& c_{2,2}{H}_G& c_{2,3}{H}_G\\ c_{3,1}{H}_B& c_{3,2}{H}_B& c_{3,3}{H}_B\end{array}\right)$

(68) where c.sub.i,j are the following coefficients:

(69) $C=\left(\begin{array}{ccc}\frac{1}{\sqrt{3}}& \frac{-1}{\sqrt{2}}& \frac{-1}{\sqrt{6}}\\ \frac{1}{\sqrt{3}}& \frac{1}{\sqrt{2}}& \frac{-1}{\sqrt{6}}\\ \frac{1}{\sqrt{3}}& 0& \frac{2}{\sqrt{6}}\end{array}\right)$

(70) Such matrix H.sub.p(p) is then used to calculate a precision matrix Q.sub.p as described in the equation (3).

(71) The matrix Q.sub.X is defined by:

(72) $Q_x=\left(\begin{array}{ccc}{D}^{T}D& 0& 0\\ 0& D^{T}D& 0\\ 0& 0& D^{T}D\end{array}\right)$$\mathrm{with}$$=0.05.$

(73) where D is the concatenation of the horizontal and vertical first order derivative operators.

(74) Then an additional triplet of chromatic images corresponding to the same light source positioned at a distance + is acquired. A precision matrix Q.sub.p(p+p) is obtained using the same method.

(75) From such matrices, a digital approximation of the equation (2) is determined with the equation:

(76) $\frac{d{Q}_P}{dp}\left(p\right)\frac{Q_P\left(p+p\right)-Q_P\left(p\right)}{p}$

(77) Such CRB describes the system analyzed for the source position p.

(78) Such calculation is repeated on a plurality of depths p, to determine, for each one of the assemblies, a curve of the CRB(p) values as a function of the distance p from the source, two examples of which are shown in FIG. 5.

(79) The digital approximation of the CRB in the position p can advantageously be calculated from a number of measures S>2 around the position p: +.sub.s, for s=1, . . . , S.

(80) The optimal system is determined by selecting the one having the best CRB curve. Such criterion can be determined according to the minimization of the average of the curve points, or the curve integrant, or more generally, any quality function deduced from the collection of the CRB values.

(81) FIG. 8 shows an exemplary selected curve and related values for the determination of the optical assembly shown in FIG. 7.

(82) Alternatively, this process can be virtually implemented from optical simulation software of the ZEMAX type.

(83) To optimize the axial chromatic aberration, the combination of two lenses having different indexes or the combination of a refractive lens and a diffractive lens can be selected. In order not to be limited by the diffraction efficiency of the diffractive lens over a wide spectral band, the combination of two optics having different constringencies is preferred.

(84) In order to subsequently correct the lateral chromatic aberration and the distortion, it is necessary to check that the point position of the image is the same in the XY plane of the detector for each wavelength and that the barycenters of the image spots for each wavelength for a point of the field are in the same position on the detector.

(85) It is, therefore, necessary to distribute the lenses on both sides of the diaphragm.

(86) The optical system meets the constraint of the telecentric feature (the average angle of the field radiuses at the outlet of the pupil and comes perpendicularly onto the detector). This property ensures that the barycenters of the image spots for each wavelength for a point in the field will be in the same position on the detector in the case of defocusing. Such constraint will be taken into account by the lens closest to the detector.

(87) The system preferably has a large opening (in the example, the system is open to F/3) so as to reduce the depth of field and accentuate the blur variations. A large opening requires a larger correction of the aberrations. To limit the number of aspheric surfaces for reasons of tolerancing, lenses are added into the optical architecture. To limit the opening of the lens positioned on either side of the diaphragm and thus reduce the amplitude of the aberrations, a duplication of each one of these lenses is provided. Another solution consists in adding a lens at the diaphragm so as to efficiently correct the spherical aberration. The system then has a three lens architecture called Cooke triplet. Such solution, however, provides no access to the diaphragm (for positioning an encoded diaphragm, for example).

(88) The preferred architecture consists of four lenses positioned on either side of the diaphragm. This architecture is a variant of the Rapid rectilinear (page 278 of the book: Handbook of Optical Systems, Volume 4) or double Gauss (page 303 of the book: Handbook of Optical Systems, Volume 4). The differences result from the absence of doublets and the constraint of telecentric features (with the fourth lens, which is closest to the detector, being spaced from the third lens to ensure the telecentric feature and the correction of the field curvature).

(89) The optimization using optical calculation software such as ZEMAX makes it possible to select the radii of curvature, the thickness and the position of each lens. The materials of each lens are so selected as to produce the correct amount of axial chromatic aberration. Conicity is added to one of the surfaces in order to correct residual aberrations.