METHOD FOR OBTAINING A POSITION OF A MAIN LENS OPTICAL CENTER OF A PLENOPTIC CAMERA

Abstract

A method is described for obtaining a position of a main lens optical center of a plenoptic camera. The plenoptic camera has a micro-lens array (MLA) positioned in front of a sensor, the main lens optical center position being defined in a referential relative to the sensor. Such method is remarkable in that it obtains, from a 4D raw light-field data of a monochromatic scene, a set of symmetry axes, each symmetry axis of the set being defined as a line associated with a micro-image, the line passing in the neighborhood of the micro-image center coordinates of the micro-image it is associated with, and in the neighborhood of the brightest pixel in the micro-image it is associated with, the set comprising at least two symmetry axes and determines the position of the main lens optical center according to at least two symmetry axes of the set.

Claims

1. A method for obtaining a position of a main lens optical center of a plenoptic camera comprising a micro-lens array (MLA) positioned in front of a sensor, said main lens optical center position being defined in a referential relative to said sensor, and wherein said method comprises: obtaining, from a 4D raw light-field data of a monochromatic scene, a set of symmetry axes, each symmetry axis of said set being defined as a line associated with a micro-image, said line passing in the neighborhood of the micro-image center coordinates of the micro-image it is associated with, and in the neighborhood of the brightest pixel in said micro-image it is associated with, said set comprising at least two symmetry axes; determining the position of the main lens optical center according to at least two symmetry axes of said set.

2. The method according to claim 1, wherein said neighborhood of the micro-image center coordinates and said neighborhood of the brightest pixel are defined according to Euclidian distance metric and a threshold.

3. The method according to claim 1, wherein said line is passing through the micro-image center coordinates of the micro-image it is associated with, and through the brightest pixel in said micro-image it is associated with.

4. The method according to claim 1, wherein it comprises determining said set of symmetry axes, said determining comprising, for a given micro-image I.sub.k having micro-image center coordinates (c.sub.x.sup.k,c.sub.y.sup.k): obtaining the M-brightest pixels {p.sub.m}.sub.m=1.sup.M comprised in I.sub.k, M being an integer greater than one; obtaining M lines, each line being defined as passing through one of said M brightest pixels and said micro-image center coordinates (c.sub.x.sup.k,c.sub.y.sup.k); determining, for each of said M lines, a number of symmetric pixel pairs s.sub.n; adding in said set of symmetry axes, the line having the largest number of symmetric pixel pairs s.sub.n among the M values of number of symmetric pixel pairs s.sub.n.

5. The method according to claim 4, wherein said determining, for each of said M lines, is done for 2.M−1 lines, said 2.M−1 lines comprising said M lines and M−1 interpolated lines, an interpolated line passing through the micro-image center coordinates (c.sub.x.sup.k,c.sub.y.sup.k) and a point comprised in said micro-image I.sub.k, said point being further comprised in a region having for border two of said M lines.

6. The method according to claim 4, wherein M is greater than five.

7. The method according to claim 4, wherein said symmetric pixel pairs are identified according to the predominant gradient direction of pixels.

8. The method according to claim 7, wherein said predominant gradient direction of pixels is obtained via the estimation of a structure tensor.

9. The method according to claim 4, wherein it comprises removing lines that are outside a circle centered on the center of the 4D raw light-field data, and having a radius R of pixels, where R is an integer smaller than 40.

10. The method according to claim 1, wherein it comprises determining said set of symmetry axes, said determining comprising, for a given micro-image I.sub.k: applying an interpolation method on said micro-image I.sub.k delivering a high-resolution micro-image; determining line parameters a, b, c defining an equation line ax+by +c=0 that minimize a sum of first and a second element, the first element being a square difference between said micro-image I.sub.k and sub-sampled said high-resolution micro-image, and the second element being a measure of symmetry of said high-resolution micro-image with regards to the line with parameters a, b, c.

11. The method according to claim 10, wherein said determining, for a given micro-image I.sub.k further comprises verifying that a distance between the 4D raw light-field data center and said line with parameters a, b, c is no larger than R pixels.

12. The method according to claim 1, wherein said determining the position of the main lens optical center is done as a function of an intersection of at least a part of the symmetry axes of said set.

13. The method according to claim 1, wherein said determining the position of the main lens optical center is done by minimizing a weighted sum of distances between each of said symmetry axis and unknown main lens optical center coordinates (x.sub.o,y.sub.o) of said plenoptic camera.

14. The method according to claim 1, wherein only micro-images positioned at the periphery of the 4D raw light-field data are used for obtaining said set of symmetry axes.

15. The method according to claim 1, wherein at least 25% of the micro-images comprised in the 4D raw light-field data are used for obtaining said set of symmetry axes.

16. The method according to claim 1, wherein all the micro-images comprised in the 4D raw light-field data are used for obtaining said set of symmetry axes.

17. A computer-readable and non-transient storage medium storing a computer program comprising a set of computer-executable instructions to implement a method for processing 4D raw light field data when the instructions are executed by a computer, wherein the instructions comprise instructions, which when executed, configure the computer to perform the method of claim 1.

18. An electronic device for obtaining a position of a main lens optical center of a plenoptic camera comprising a micro-lens array (MLA) positioned in front of a sensor, said main lens optical center position being defined in a referential relative to said sensor, and wherein said electronic device comprises a memory unit and at least one processor coupled to said memory unit, the at least one processor being configured to: obtain, from a 4D raw light-field data of a monochromatic scene, a set of symmetry axes, each symmetry axis of said set being defined as a line associated with a micro-image, said line passing in the neighborhood of the micro-image center coordinates of the micro-image it is associated with, and in the neighborhood of the brightest pixel in said micro-image it is associated with, said set comprising at least two symmetry axes; and determine the position of the main lens optical center according to at least two symmetry axes of said set.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and other aspects of the invention will become more apparent by the following detailed description of exemplary embodiments thereof with reference to the attached drawings in which:

[0061] FIG. 1 presents schematically the main components comprised in a plenoptic camera that enables the acquisition of light field data on which the present technique can be applied;

[0062] FIG. 2 presents an image captured by a sensor array of FIG. 1;

[0063] FIG. 3 presents how the optical center and each micro-image center lie on a line. The symmetry axes of all the micro-images intersect at the optical center;

[0064] FIG. 4 is a flowchart that roughly depicts the main steps of the method for estimating the optical center position of a plenoptic camera, according to one embodiment of the disclosure;

[0065] FIG. 5 emphasizes on the main steps of the detection step mentioned in FIG. 4;

[0066] FIG. 6 emphasizes on the main steps of the interpolation step mentioned in FIG. 5;

[0067] FIG. 7(a) is a flowchart of the symmetric pixel pair identification from structure tensor used in one embodiment of the disclosure;

[0068] FIG. 7(b) is an illustration of the construction of 3×3 windows around pixel i and pixel j respectively;

[0069] FIG. 8 is a flowchart that depicts the main step of the symmetry lines/axes detection in one embodiment of the disclosure;

[0070] FIG. 9 is a flowchart that describes the main step of the combination/use of the symmetric axes/lines for estimating the position of the optical center, according to one embodiment of the disclosure;

[0071] FIG. 10 corresponds to a graph that details the relationship between percentage of used micro-images in the estimation method, and the estimation error (at focal length 8 mm with the standard deviation of noise set as 0.1) of the position of the main lens optical center; and

[0072] FIG. 11 presents an example of device that can be used to perform one or several steps of methods disclosed in the present document.

DETAILED DESCRIPTION

[0073] The FIG. 1 presents schematically the main components comprised in a plenoptic camera that enables the acquisition of light field data on which the present technique can be applied.

[0074] More precisely, a plenoptic camera comprises a main lens referenced 101, and a sensor array [i.e., an array of pixel sensors (for example a sensor based on CMOS technology)], referenced 104. Between the main lens 101 and the sensor array 104, a micro-lens array referenced 102, that comprises a set of micro-lenses referenced 103, is positioned. It should be noted that optionally some spacers might be located between the micro-lens array around each lens and the sensor to prevent light from one lens to overlap with the light of other lenses at the sensor side. In one embodiment, all the micro-lenses have the same focal. In another embodiment, the micro-lens can be classified into at least three groups of micro-lenses, each group being associated with a given focal, different for each group. Moreover, in a variant, the focal of a micro-lens is different from the ones positioned at its neighborhood; such configuration enables the enhancing of the plenoptic camera's depth of field. It should be noted that the main lens 101 can be a more complex optical system as the one depicted in FIG. 1 (as for example the optical system described in FIGS. 12 and 13 of document GB2488905) Hence, a plenoptic camera can be viewed as a conventional camera plus a micro-lens array set just in front of the sensor as illustrated in FIG. 1. The light rays passing through a micro-lens cover a part of the sensor array that records the radiance of these light rays. The recording by this part of the sensor defines a micro-lens image.

[0075] The FIG. 2 presents an image captured by the sensor array 104. Indeed, in such view, it appears that the sensor array 104 comprises a set of pixels, referenced 201. The light rays passing through a micro-lens cover a number of pixels 201, and these pixels record the energy value of light rays that are incident/received.

[0076] Hence the sensor array 104 of a plenoptic camera records an image which comprises a collection of 2D small images (i.e. the micro-images referenced 202) arranged within a 2D image (which is also named a raw 4D light-field image). Indeed, each small image (i.e. the micro-images) is produced by a micro-lens (the micro-lens can be identified by coordinates (i,j) from the array of lens). Hence, the pixels of the light-field are associated with 4 coordinates (x,y,i,j). L(x,y,i,j) being the 4D light-field recorded by the sensor illustrates the image which is recorded by the sensor. Each micro-lens produces a micro-image represented by a circle (the shape of the small image depends on the shape of the micro-lenses which is typically circular). Pixel coordinates (in the sensor array) are labelled (x,y). p is the distance between two consecutive micro-images, p is not necessary an integer value. Micro-lenses are chosen such that p is larger than a pixel size δ. Micro-images are referenced by their coordinates (i,j). Each micro-image samples the pupil of the main-lens with the (u,v) coordinate system. Some pixels might not receive any photons from any micro-lens especially if the shape of the micro-lenses is circular. In this case, the inter micro-lens space is masked out to prevent photons to pass outside from a micro-lens, resulting in some dark areas in the micro-images. If the micro-lenses have a square shape, no masking is needed. The vignetting of the optical system is another reason for which some pixels may not receive any photons. The center of a micro-image (i,j) is located on the sensor at the coordinates (x.sub.i,j,y.sub.1,j). θ is the angle between the square lattice of pixel and the square lattice of micro-lenses, in FIG. 2 θ=0. Assuming the micro-lenses are arranged according to a regular square lattice, the (x.sub.i,j,y.sub.i,j) can be computed by the following equation considering (x.sub.0,0, y.sub.0,0) the pixel coordinates of the micro-image (0,0):

[00001] $[\begin{matrix} x_{i, j} \\ y_{i, j} \end{matrix}] = p [\begin{matrix} \cos .Math. .Math. θ & - \sin .Math. .Math. θ \\ \sin .Math. .Math. θ & \cos .Math. .Math. θ \end{matrix}] [\begin{matrix} i \\ j \end{matrix}] + [\begin{matrix} x_{0, 0} \\ y_{0, 0} \end{matrix}]$

[0077] FIG. 2 also illustrates that an object point from the scene is visible on several contiguous micro-images (dark dots). In one embodiment, the distance between 2 consecutive views of an object is w, this distance is named the replication distance. Hence, an object is visible on r consecutive micro-images with:

[00002] $r = .Math. \frac{p}{.Math. p - w .Math.} .Math.$

r is the number of consecutive micro-images in one dimension. An object is visible in r.sup.2 micro-images. Depending on the shape of the micro-lens image, some of the r.sup.2 views of the object might be invisible.

[0078] More details related to plenoptic camera can be found out in the Section 4 entitled “Image formation of a Light field camera” in the article entitled “The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution” by Tom E. Bishop and Paolo Favaro, published in the IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, No 5, in May 2012.

[0079] It should be noted that the present technique can also be applied on “conventional camera” (in the sense that no additional micro-lens array is positioned between the main lens and array of pixels), in the case that at least a part of the pixels of such conventional camera are designed in the same way (or similar way) as the one described in the document US2013258098. Indeed, document US2013258098 discloses a pixel that can record light field data due to the use of several light receiving sections (for example referenced 116 and 117 in document US2013258098). Hence, one skilled in the art could assimilate such conventional camera with an array of pixels integrating the technique of document US2013258098 as a kind of plenoptic camera as depicted in FIG. 1, in which each micro-lens concentrates light rays on two pixels comprised in the sensor 104. It should be noted that technique of document US2013258098 can be generalized in the sense that a pixel can record more than two data information (obtained by the two low and high receiving sections), if more receiving section are integrated in the architecture of a pixel. The present disclosure can be used on raw images of “conventional camera” integrating pixels that can record light field data as mentioned previously. Indeed, these raw images can be assimilated to a set of micro-images.

[0080] It should also be noted that the present disclosure can also be applied to other devices that acquire 4D light field data such as devices that comprise coded aperture elements as depicted in document US 2010/0265386, or in the article entitled “Image and depth from a conventional camera with a coded aperture” by A. Levin a al., published in the proceedings of SIGGRAPH 2007, or use wavefront coding techniques as mentioned in the article entitled “Extended depth of field through wave-front coding” by Edward R. Dowski, Jr., and W. Thomas Cathe, published in Applied Optics, 1995 Apr. 10.

[0081] The proposed technique relates to a method for obtaining/estimating a position of a main lens optical center of a plenoptic camera. In one embodiment of the disclosure, such technique uses a 4D raw light-field data of a white scene and a set of micro-images center (c.sub.x.sup.k, c.sub.y.sup.k) as inputs. In one embodiment of the disclosure, only the micro-images located/positioned at the periphery of the 4D raw light-field data are used for performing the estimation of the position of the main lens optical center of the plenoptic camera.

[0082] It should be noted that we assume that the micro-images center (c.sub.x.sup.k, c.sub.y.sup.k) associated with a micro-image I.sub.k is estimated by any technique of the state-of-the-art (as the one described in the article entitled “Decoding, calibration and rectification for lenselet-based plenoptic cameras” by Dansereau et al., published in the conference proceedings of Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE).

[0083] The proposed technique relies on the symmetry characteristics of micro-images for estimating the optical center. Indeed, each micro-image is reflection symmetric (i.e., symmetry with respect to the line connecting the micro-image center, and the optical center, as shown partially in FIG. 3), because micro-images result from the circular construction of the main-lens aperture and the micro-lens apertures. In theory, the intersection of two circles is reflection symmetric and the symmetry axis is the line passing through the centers of the corresponding circles. This is described in the book entitled “Computer Algebra and Geometric Algebra with Applications” 6th international workshop IWMM, pages 356-357, 2004. Then theoretically the symmetry axes of all the micro-images intersect at one point: the optical center.

[0084] The FIG. 4 is a flowchart that roughly depicts the main steps of the method for estimating the optical center position of a plenoptic camera according to one embodiment of the disclosure. In a step referenced 401, an electronic device obtains a 4D raw light-field data of a white scene (captured/generated by a plenoptic camera) and a set of micro-lens images center (c.sub.x.sup.k,c.sub.y.sup.k). Such set can comprise all or only a part of micro-images center compared to the whole of micro-images comprised in the 4D raw light-field data. In one embodiment of the disclosure, the electronic device in the step 401 detects the symmetry axes/lines of all the micro-images and then, in a step referenced 402, computes their intersection point as the optical center. In a variant, only a part of the micro-images are used in the detecting step 401. In one embodiment of the disclosure, at least 25% of the micro-images are used in such detecting step 401. In another embodiment, only the micro-images positioned in the periphery of the 4D raw light-field data are used in the step 401. Indeed, due to the cat's eye vignetting that occur in the micro-images at the periphery of the 4D raw light-field data, it is easier to detect a symmetry for that kind of micro-images compared to the ones located around the center of the 4D raw light-field data (that can numerically have a lot of symmetry axis).

[0085] More generally, any line-symmetry detection method can be used to provide a trade-off between the complexity and the precision in the step 401. For example, in one embodiment of the disclosure, an interpolation-based method is used, where micro-images are interpolated to some scale and the reflection symmetry axes are determined using a designed method. This is computation-efficient, but the precision is limited, as the scale of interpolation is limited. In another embodiment, an optimization based method is used to further improve the precision. The details of both embodiments are detailed in the following. It should be noted that the effect of the Bayer pattern (in the 4D raw light-field data) is corrected before the symmetry detection is done. In one embodiment, pixel intensities are multiplied with known white balance gain parameters to consider the effect of the Bayer pattern.

[0086] In one embodiment of the disclosure, following the symmetry detection 401, the optical center is determined by computing 402 the intersection point of the determined symmetry axes. In practice, due to the accuracy of symmetry detection and contamination of noise and Bayer pattern, the detected symmetry axes may not intersect at one point. Therefore, in another embodiment of the disclosure, the optical center could be determined by determining a point that has the smallest total distance to all or a part of the symmetry axes/lines (that could also be weighted; for example, the symmetry axes/lines associated with the micro-images that are located at the periphery of the 4D raw light-field data could be considered as more relevant/important for the optimization process and therefore weighted more heavily than the ones from the center of the 4D raw light-field data).

[0087] It should be noted that, in one embodiment of the disclosure, once a set of symmetry axes/lines is obtained from step 401, and before executing the step 402, the electronic device can perform an elimination step that is aim to eliminate the symmetry axes/lines that have invalid direction. It should be noted that such elimination step can be also integrated into the step 401 for eliminating some symmetry lines associated with some micro-images.

[0088] Hence, in one embodiment of the disclosure, after a symmetry axis is detected for each micro-image (or a part of the micro-images), those symmetry axes with invalid directions are eliminated using the prior knowledge that the optical center resides in a circular neighborhood of the center of the raw image (i.e. the 4D raw light-field data) with a radius R>0 of pixels (with R being equal for example to 40 pixels). If the distance from the image center to a determined symmetry axis is larger than R, then the symmetry axis is not taken into consideration when the final estimation of the optical axis is performed in step 402.

[0089] The FIG. 5 emphasizes on the main steps of the detection step mentioned in FIG. 4.

[0090] More precisely, in one embodiment of the disclosure, the step 401 determines for each micro-image I.sub.k or a selection of a set of micro-image I.sub.k comprised in the 4D raw light-field data (with the corresponding micro-image center coordinates (c.sub.x.sup.k, c.sub.y.sup.k) available), the symmetry axis I.sub.k as follows:

[0091] In a step referenced 501, the electronic device determines a set of N candidate symmetric axes/lines {I.sub.n}.sub.n=1.sup.N for a current micro-image. The symmetry axis should go through the brightest pixel in the micro-image (due to the radial symmetry of the circular main-lens and micro-lens) and the center of the micro-image. In practice, however, the brightest pixel in the micro-image might not be the actual brightest one due to noise and/or Bayer pattern. To circumvent this, it is proposed to determine the M brightest pixels {p.sub.m}.sub.m=1.sup.M in the current micro image I.sub.k (by analyzing and comparing the different intensity values of the pixels comprised in the current micro-image). These M points p.sub.m are then used along with the micro-image center coordinates (c.sub.x.sup.k, c.sub.y.sup.k) to form the corresponding M candidate symmetry axes/lines. In one embodiment, we can have N which is equal to M.

[0092] In an optional step referenced 502, the electronic device interpolates the (M−1) lines obtained from step 501 in order to obtain N (N=2M−1) candidate symmetry axes/lines {l.sub.n}.sub.n=1.sup.N that lie between the M candidate lines for improving the estimation precision of the optical center. Details of the interpolation process are depicted in FIG. 6.

[0093] In a step referenced 503, for each candidate symmetry axis l.sub.n (among the N=M or N=2M−1 candidate lines depending on the execution of the step 502), the number of symmetric pixel pairs s.sub.n with regards to the line l.sub.n are determined. The symmetric pixel pairs are identified based on structure tensor, and sub-pixel accuracy is considered in the process for improved precision. Indeed, as it is crucial to identify symmetric pixel pairs correctly in order to measure candidate symmetry axes/lines, a robust algorithm based on structure tensor for the identification is proposed in one embodiment of this disclosure. Details of the determination of the number of symmetric pixel pairs for a given line are described in FIGS. 7(a) and 7(b). Then the candidate symmetry axis/line with the largest s.sub.n is determined as the symmetry axis of the micro-image l.sub.k.

[0094] It should be noted that instead of using all the micro-images in the white image for symmetry detection, which is computationally expensive, only micro-images at the periphery of the white image can be used to reduce computation complexity with little effect on the accuracy of the final optical center estimation. This is because compared to micro-images in the center, symmetry detection is more accurate for micro-images at the periphery that are only reflection symmetric in terms of shape due to cat's eye vignetting.

[0095] The FIG. 6 emphasizes the main steps of the interpolation step 502 mentioned in FIG. 5.

[0096] At the output of step 501, the electronic device has determined a set of N=M candidate symmetry axes/lines. Each candidate symmetry axis l.sub.n is represented as a.sub.nx+b.sub.ny+c.sub.n=0. The parameters a.sub.m, b.sub.m, c.sub.m for candidate lines that connect M brightest pixels {p.sub.m}.sub.m=1.sup.M and (c.sub.x.sup.k, c.sub.y.sup.k) are computed as follows if {p.sub.m}.sub.m=1.sup.M have different x-coordinates from c.sub.x.sup.k (i.e., l.sub.m is not vertical):

[00003] $a_{m} = - \frac{c_{y}^{k} - y_{p_{m}}}{c_{x}^{k} - x_{p_{m}}}, .Math. b_{m} = 1, .Math. c_{m} = - a_{m} .Math. c_{x}^{k} - b_{m} .Math. c_{y}^{k} .$

Otherwise, a.sub.m, b.sub.m, c.sub.m are computed as follows:

[00004] $a_{m} = 1, .Math. b_{m} = 0, .Math. c_{m} = - c_{x}^{k} .$

In the case that an interpolation occurs, the electronic device determines, in a step referenced 601, interpolated lines, where an interpolated line l.sub.n that lies in for example between l.sub.m-1 and l.sub.m has parameters that are determined as follows if b.sub.m-1≠0 and b.sub.m≠0:

[00005] $a_{n} = \frac{a_{m - 1} + a_{m}}{2}, .Math. b_{n} = 1, .Math. c_{n} = - a_{n} .Math. c_{x}^{k} - b_{n} .Math. c_{y}^{k} .$

[0097] Otherwise, a.sub.n=2a.sub.m if b.sub.m-1=0, and a.sub.n=2a.sub.m-1 if b.sub.m=0. The computation of b.sub.n and c.sub.n remains the same.

[0098] The electronic device executes the step 601 until it has obtain a given number of candidate symmetry axes/lines. In one embodiment of the disclosure, such given number can be equal to 2M−1.

[0099] The FIG. 7(a) is a flowchart of the symmetric pixel pair identification from structure tensor used in one embodiment of the disclosure in step 501.

[0100] As micro-images may suffer from noise, it is not robust to identify symmetric pixel pairs just by their intensity values as mentioned in the description of step 501. In order to attenuate the effect of noise, in one embodiment a window around each pixel is considered to take neighboring pixels into consideration, and two pixels are determined as a symmetric pixel pair with regards to l.sub.n if the predominant gradient directions of the windows formed around them are symmetric with regards to the line l.sub.n, as illustrated in FIG. 7(b). Indeed, FIG. 7(b) is an illustration of the construction of 3×3 windows around pixel i and pixel j respectively. The line referenced 710 is one candidate symmetry axis of the micro-image, and the lines referenced 720 and 730, with arrows represent the predominant gradient directions of the windows.

[0101] The predominant gradient direction of a window of pixels 1 is estimated by structure tensor. The structure tensor, also referred to as the second-moment matrix, is a matrix derived from the gradient of I. It summarizes the predominant directions of the gradient in I. The structure tensor matrix T for I is defined as follows:

[00006] $T = [\begin{matrix} I_{x}^{2} & I_{x} .Math. I_{y} \\ I_{x} .Math. I_{y} & I_{y}^{2} \end{matrix}],$

where I.sub.x and I.sub.y are the partial derivatives of 1 with regards to the x and y axes, respectively. T has two eigenvectors ν.sub.1, ν.sub.2, with corresponding eigenvalues γ.sub.1, γ.sub.2 sorted in descending order (i.e., γ.sub.1≧γ.sub.2). If γ.sub.1>γ.sub.2, then φ=ν.sub.1 gives the predominant gradient direction. If γ.sub.1=γ.sub.2, then I is isotropic (e.g., constant) and thus there is no predominant gradient.

[0102] Having estimated the predominant direction φ.sub.i and φ.sub.j for each pixel pair {i,j} with coordinates symmetric with regards to l.sub.n, the angle between the predominant directions and the direction of I.sub.n is then computed [(θ.sub.i,θ.sub.j) for pixel pair {i,j}], as shown in FIG. 7(b). Pixels i and j contain symmetric intensity information with respect to l.sub.n if θ.sub.i, θ.sub.j are close to each other in value, i.e.,

|θ.sub.i−θ.sub.j|<ε,

where ε is a very small positive number.

[0103] In one embodiment of the disclosure, ε is equal to 5 degrees.

[0104] Further, when pixel i with integer coordinates is considered, pixel j that is symmetric with pixel i with regards to l.sub.n might not have integer coordinates. It is proposed here that the corresponding intensities for the non-integer coordinates of all of the window pixels are calculated by means of bilinear interpolation.

[0105] Therefore, the electronic device for a given micro-image obtains as input a set of symmetry candidate lines.

[0106] Then, in a step referenced 701, the electronic device determines a window size to be used when processing the pixels in the micro-image. The window can have the shape of a square or a rectangular.

[0107] Then, for a given line, in a step referenced 702, the electronic device determines for each pair of symmetric pixels with regards to such given line the predominant gradient direction of the pair of windows pixels. It should be noted that, in one embodiment, the electronic determines the predominant gradient direction based on pair of symmetric points. Here a pair of corresponding points means two points whose coordinates are symmetric with respect to the given line. For each pixel p on one side of the line, the coordinate of its corresponding point q is computed according to the two equations on page 23 (and such point q is not necessary a pixel as the corresponding point of a pixel might be located at a subpixel position). Then the predominant gradient directions of the pixel patches centered at p and q respectively are computed.

[0108] Then, for the same given line, in a step referenced 703, the electronic device determines, for each pair of symmetric pixels/points, the difference between the angles (defined by the predominant direction and the given line). If the difference is below a threshold, a pair of symmetric pixels can be qualified as such, and therefore the value s.sub.n associated with the given line l.sub.n is incremented by one.

[0109] The step 702 and 703 are done for all the candidate lines.

[0110] It should be noted that, in the case that step 502 is done, while more interpolated candidate symmetry axes may lead to more accurate estimation of the optical center, it is uncertain how many lines need to be interpolated. Also, it is time-consuming to interpolate a large number of lines and compute the number of symmetric pixel pairs by structure tensor with bilinear interpolation of non-integer window coordinates.

[0111] Hence, in one embodiment, the step 401 that performs symmetry detection is formulated as an optimization problem so that the symmetry axis can be more efficiently detected directly by solving the optimization problem. The optimization-based symmetry detection algorithm corresponds to another embodiment of symmetry axes detection.

[0112] The FIG. 8 is a flowchart that depicts the main step of the symmetry lines/axes detection in one embodiment of the disclosure.

[0113] In such embodiment, an optimization-based method for symmetry detection is designed in order to achieve even higher precision than the interpolation-based method described in FIG. 5. The idea is to try to find the symmetry axis of a micro-image that has arbitrarily higher resolution than the captured micro-image. In particular, the optimization aims to:

[0114] 1) find a high-resolution version Î of the captured micro-image I, and

[0115] 2) determine the symmetry axis l(a, b, c) of the high-resolution micro-image Î, with the symmetry axis l represented as ax+by +c=0.

[0116] The objective is a weighted sum of

[0117] 1) the data fidelity term meaning the squared difference between the observed micro-image I and the sub-sampled instance of the high resolution image Î where the operator H performs the sub-sampling, and

[0118] 2) a measure of symmetry of I with regards to the line l(a, b, c), denoted by E.sub.sym(Î,l(a, b, c)), where smaller value means more symmetric pixel pairs. Besides, in one embodiment, a constraint is added enforcing the distance d between the sensor center C.sub.o=(x.sub.o, y.sub.o) to be determined and l(a, b, c) to be no larger than R pixels. Hence the problem is formulated as follows:

[00007] $\min_{z, a, b, c} .Math. {.Math. H .Math. \hat{I} - I .Math.}_{2}^{2} + ρ .Math. .Math. E_{sym} (\hat{I}, l (a, b, c))$ $s . t . .Math. d \leq R$ $where$ $d = \frac{.Math. {ax}_{o} + {by}_{o} + c .Math.}{\sqrt{a^{2} + b^{2}}}$

The symmetry function E.sub.sym(Î, l(a, b, c)) is defined as the sum of squared difference between the intensity value of pixel coordinates p=(x.sub.p,y.sub.p) and =(x.sub.q, y.sub.q), where q is the coordinate of the point that has the same distance to l(a, b, c), and resides at the reflectionally symmetric coordinate with respect to p). q is computed as follows:

[00008] $x_{q} = x_{p} - \frac{2 .Math. a ({ax}_{p} + {by}_{p} + c)}{a^{2} + b^{2}}, .Math. y_{q} = y_{p} - \frac{2 .Math. b ({ax}_{p} + {by}_{p} + c)}{a^{2} + b^{2}} .$

[0119] Then the symmetry function is

[00009] $E_{sym} (\hat{I}, l (a, b, c)) = \underset{p}{.Math.} .Math. {({\hat{I}}_{p} - {\hat{I}}_{q})}^{2} .$

[0120] In order to solve the optimization problem with two variable sets {Î} and {a, b, c}, in one embodiment an alternating minimization algorithm is used to determine one variable set at a time while the other is fixed. Hence, the electronic device has to perform for each micro-image (or at least a part of selected micro-images) the following steps:

[0121] In a step referenced 801, the electronic device initializes Î by upsampling I by a scaling factor s using an interpolation method (e.g., bilinear interpolation).

[0122] In a step referenced 802, the electronic device optimizes the corresponding {a, b, c} by applying an optimization method (e.g., Kovesi's algorithm).

[0123] In a step referenced 803, the electronic device fixes the estimated {a, b, c} from the previous step, and update z by any optimization method (e.g., quadratic programming).

[0124] In a step referenced 804, the electronic device executes the step 802 until a predefined level of convergence is reached. An example is when the data term becomes smaller than a predefined value.

[0125] Note that a larger scaling factor s leads to higher accuracy but more computation complexity. The choice of the scaling factor s depends on the accuracy and complexity requirement of a specific application.

[0126] In general, the algorithm converges within 5 iterations. Thus the computation complexity of this algorithm is less than 5 times of that of the previous interpolation-based method.

[0127] The FIG. 9 is a flowchart that describes the main step of the combination/use of the symmetry axes/lines for estimating the position of the optical center.

[0128] In one embodiment of the disclosure, the electronic device obtains a set of symmetry lines/axes associated with a set of micro-images, and a set of weight values that are associated to each of the symmetry lines of the set of symmetry lines. Indeed, the weigh values aim at given more importance to some symmetry lines compared to others (for example, the symmetry lines from the micro-images positioned/located at the periphery of the 4D raw Light Field data could be associated with weight values that are more important than the weight values associated with symmetry lines of micro-images located close to the center of the 4D raw Light Field data).

[0129] Then, in a step referenced 901, the electronic device solves an optimization problem that provides the estimation of the optical center.

[0130] Indeed, as discussed earlier, all the symmetry axes/lines are not guaranteed to intersect at one point. Hence, in one embodiment, the final optical center C.sub.o=(x.sub.o, y.sub.o) is determined by minimizing the sum of weighted distances between C.sub.o=(x.sub.o, y.sub.o) and each symmetry axis l.sub.k, to find the most probable optical center, i.e.,

[00010] $D = {.Math.}_{k = 1}^{K} .Math. w_{k} .Math. \frac{.Math. a_{k} .Math. x_{o} + b_{k} .Math. y_{o} + c_{k} .Math.}{\sqrt{a_{k}^{2} + b_{k}^{2}}}$

where w.sub.k is the weighting parameter for the k-th symmetry axis. In one embodiment, w.sub.k is large when the corresponding micro-image is at the periphery as its symmetry detection tends to be more accurate.

[0131] In practice, it is difficult to minimize D when it is defined as a function of the absolute values. In order to make the problem tractable, D is replaced by D′, which is formulated as the sum of squared distance:

[00011] $D^{'} = {.Math.}_{k = 1}^{K} .Math. w_{k} .Math. \frac{{(a_{k} .Math. x_{o} + b_{k} .Math. y_{o} + c_{k})}^{2}}{a_{k}^{2} + b_{k}^{2}}$

In order to minimize D′ for estimation of x.sub.o and y.sub.o, in one embodiment the partial derivatives of D′ with respect to x.sub.o and y.sub.o are considered respectively and the derivatives are set to 0. This results in the following equation (w.sub.k is set to 1 for simplicity here):

[00012] $[\begin{matrix} \underset{k}{.Math.} .Math. \frac{a_{k}^{2}}{a_{k}^{2} + b_{k}^{2}} & \underset{k}{.Math.} .Math. \frac{a_{k} .Math. b_{k}}{a_{k}^{2} + b_{k}^{2}} \\ \underset{k}{.Math.} .Math. \frac{a_{k} .Math. b_{k}}{a_{k}^{2} + b_{k}^{2}} & \underset{k}{.Math.} .Math. \frac{b_{k}^{2}}{a_{k}^{2} + b_{k}^{2}} \end{matrix}] [\begin{matrix} x_{o} \\ y_{o} \end{matrix}] = - [\begin{matrix} \underset{k}{.Math.} .Math. \frac{a_{k} .Math. c_{k}}{a_{k}^{2} + b_{k}^{2}} \\ \underset{k}{.Math.} .Math. \frac{b_{k} .Math. c_{k}}{a_{k}^{2} + b_{k}^{2}} \end{matrix}]$

Then in one embodiment, the electronic device obtains the value of (x.sub.o, y.sub.o) by solving the above equation, multiplying both sides of the equation by the inverse matrix of the matrix

[00013] $[\begin{matrix} {.Math.}_{k} .Math. \frac{a_{k}^{2}}{a_{k}^{2} + b_{k}^{2}} & {.Math.}_{k} .Math. \frac{a_{k} .Math. b_{k}}{a_{k}^{2} + b_{k}^{2}} \\ {.Math.}_{k} .Math. \frac{a_{k} .Math. b_{k}}{a_{k}^{2} + b_{k}^{2}} & {.Math.}_{k} .Math. \frac{b_{k}^{2}}{a_{k}^{2} + b_{k}^{2}} \end{matrix}] .$

[0132] The FIG. 10 corresponds to a graph that details the relationship between percentage of used micro-images in the estimation method, and the estimation error (at focal length 8 mm with the standard deviation of noise set as 0.1) of the position of the main lens optical center.

[0133] It should be noted that for obtaining such graph, the micro-images are chosen from the periphery of the image, and then increased towards the center. The estimation error decreases as more micro-images are used, but the decreasing rate gets smaller. This means that more micro-images help improve the estimation accuracy, and the benefit is significant when around 50% of micro-images are used and gets smaller afterwards.

[0134] The FIG. 11 presents an example of device that can be used to perform one or several steps of methods disclosed in the present document.

[0135] Such device referenced 1100 comprises a computing unit (for example a CPU, for “Central Processing Unit”), referenced 1101, and one or more memory units (for example a RAM (for “Random Access Memory”) block in which intermediate results can be stored temporarily during the execution of instructions a computer program, or a ROM block in which, among other things, computer programs are stored, or an EEPROM (“Electrically-Erasable Programmable Read-Only Memory”) block, or a flash block) referenced 1102. Computer programs are made of instructions that can be executed by the computing unit. Such device 1100 can also comprise a dedicated unit, referenced 1103, constituting an input-output interface to allow the device 1100 to communicate with other devices. In particular, this dedicated unit 1103 can be connected with an antenna (in order to perform communication without contacts), or with serial ports (to carry communications “contact”). It should be noted that the arrows in FIG. 11 signify that the linked unit can exchange data through buses for example together.

[0136] In an alternative embodiment, some or all of the steps of the method previously described, can be implemented in hardware in a programmable FPGA (“Field Programmable Gate Array”) component or ASIC (“Application-Specific Integrated Circuit”) component. One skilled in the art could implement some of all of the steps by applying the recommendations/techniques described in the document “Reconfigurable Computing: The Theory and Practice of FPGA-Based Computing” by S. Hauck and A. DeHon, Morgan Kaufmann, 2008, or in the document “Reconfigurable Computing: From FPGAs to Hardware/Software Codesign” by Cardoso, João M. P.; Hübner, Michael, published by Sringer in 2011.

[0137] In an alternative embodiment, some or all of the steps of the method previously described, can be executed on an electronic device comprising memory units and processing units as the one disclosed in the FIG. 11.

[0138] In one embodiment of the disclosure, the electronic device depicted in FIG. 11 can be comprised in a camera device that is configure to capture images (i.e. a sampling of a light field). These images are stored on one or more memory units. Hence, these images can be viewed as bit stream data (i.e. a sequence of bits). Obviously, a bit stream can also be converted on byte stream and vice versa.

[0139] In one embodiment of the disclosure, the information related to the position of the main lens optical center of a plenoptic camera, that is obtained via the proposed technique, is stored within a memory unit comprised in the plenoptic camera. Hence, in one embodiment of the disclosure, it is proposed a plenoptic camera comprising a memory unit that stores a position of the main lens optical center of a plenoptic camera. Hence, image processing techniques executed by the plenoptic camera can benefit from the use of the information related to the position of the main lens optical center of the plenoptic camera, by reading or accessing a memory unit.

METHOD FOR OBTAINING A POSITION OF A MAIN LENS OPTICAL CENTER OF A PLENOPTIC CAMERA

Inventors

Cpc classification

Classification Explorer

G06T2200/21

PHYSICS

Classification Explorer

G02B3/0056

PHYSICS

Classification Explorer

G06T7/66

PHYSICS

Classification Explorer

G06T7/68

PHYSICS

Classification Explorer

G06T2207/10012

PHYSICS

Classification Explorer

H04N23/80

ELECTRICITY

Classification Explorer

G06T3/4007

PHYSICS

Classification Explorer

G06T7/80

PHYSICS

Classification Explorer

G06T2207/10052

PHYSICS

International classification

Classification Explorer

G06T7/68

PHYSICS

Classification Explorer

G06T3/40

PHYSICS

Classification Explorer

G02B3/00

PHYSICS

Classification Explorer

G06T7/66

PHYSICS

Abstract

Claims

Description