Method for rendering a final image from initial images acquired by a camera array, corresponding device, computer program product and computer-readable carrier medium

Abstract

A method for rendering an image, called a final image, from at least one image acquired by a camera array, is provided. According to such a method, the determination of a color value for at least one pixel of the final image, called a current pixel, comprises: for at least one initial image acquired by the camera array, obtaining a color value of a pixel associated with said current pixel within said at least one initial image, acquiring at least one color value called a real color value; computing at least one interpolated color value, from said at least one real color value; determining the color value for said current pixel, as a function of said at least one real color value and said at least one interpolated color value.

Claims

1. A method comprising: receiving at least an initial image acquired by a camera array, said initial image having a real color value and at least one associated pixel; obtaining a color value of a pixel located in a vicinity of said pixel associated with said real color value; interpolating the values of said pixel of said color value and said neighboring pixel to determine a current pixel to be used in a final rendering, said current pixel having sharp contours and colors; and performing an angular interpolation of the color values of the real color value and the neighbor color value and their associated pixels to determine a blurred area for said final rendering; and rendering a final image with sharp and blurred areas from said interpolations and calculation of said current pixel.

2. The method of claim 1, wherein said current pixel is an out-of-focus pixel, belonging to a blurred region of said final image.

3. The method of claim 1, wherein computing at least one interpolated color value from said at least one real color value comprises performing a nearest neighbors interpolation.

4. The method of claim 2, wherein computing at least one interpolated color value consists in computing a number of interpolated color values, said number being a function of a difference between a depth of said current pixel and a depth of an in-focus region of said final image.

5. The method of claim 4, wherein said number of interpolated color values increases as said difference between a depth of said current pixel and a depth of an in-focus region of said final image increases.

6. The method of claim 1, wherein said at least one real color value and said at least one interpolated color value used to determine the color value for said current pixel are determined as a function of a predetermined blur shape.

7. The method of claim 1, wherein determining the color value for said current pixel comprises determining an arithmetic average of said at least one real color value and said at least one interpolated color value.

8. The method of claim 1, wherein determining the color value for said current pixel comprises determining a weighted average of said at least one real color value and said at least one interpolated color value.

9. A device for rendering an image, called a final image, from at least one image acquired by a camera array, wherein said device comprises, for the determination of a color value for at least one pixel of the final image, called a current pixel: a module for receiving at least an initial image acquired by a camera array, said initial image having a real color value and at least one associated pixel; a module for obtaining, obtaining a color value of a pixel located in a vicinity of said pixel associated with real color value; a module for interpolating the values of said pixel of said color value and said neighboring pixel to determine a current pixel to be used in a final rendering, said current pixel having sharp contours and colors; and a module for obtaining a color value of a pixel located in a vicinity of said pixel associated with said current pixel to be used in a final rendering; said current pixel having sharp contours and colors; and a module for performing an angular interpolation of the color values of the real color value and the neighbor color value and their associated pixels to determine a blurred area for said final rendering; and a module for rendering a final image with sharp and blurred areas from said interpolations and calculation of said current pixel.

10. A non-transitory computer program product downloadable from a communication network and/or recorded on a medium readable by a computer and/or executable by a processor, comprising program code instructions for implementing a method comprising: receiving at least an initial image acquired by a camera array, said initial image having a real color value and at least one associated pixel; obtaining a color value of a pixel located in a vicinity of said pixel associated with said real color value; interpolating the values of said pixel of said color value and said neighboring pixel to determine a current pixel to be used in a final rendering, said current pixel having sharp contours and colors; performing an angular interpolation of the color values of the real color value and the neighbor color value and their associated pixels to determine a blurred area for said final rendering; and rendering a final image with sharp and blurred areas from said interpolations and calculation of said current pixel.

11. A non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for implementing a method comprising: receiving at least an initial image acquired by a camera array, said initial image having a real color value and at least one associated pixel; obtaining a color value of a pixel located in a vicinity of said pixel associated with said real color value; interpolating the values of said pixel of said color value and said neighboring pixel to determine a current pixel to be used in a final rendering, said current pixel having sharp contours and colors; performing an angular interpolation of the color values of the real color value and the neighbor color value and their associated pixels to determine a blurred area for said final rendering; and rendering a final image with sharp and blurred areas from said interpolations and calculation of said current pixel.

Description

4. BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the present disclosure can be better understood with reference to the following description and drawings, given by way of example and not limiting the scope of protection, and in which:

(2) FIGS. 1a and 1b, already described, present a simplified example of structure of a camera array that may be used to obtain the initial images used to render a final image, according to an embodiment of the present disclosure;

(3) FIG. 2 is a schematic illustration for explaining how some color values are obtained within initial images acquired by a camera array, and averaged to render a final refocused image, according to a prior art synthetic aperture refocusing technique.

(4) FIG. 3 is a flow chart for illustrating the general principle of the proposed technique for rendering a final image from at least one image acquired by a camera array, according to an embodiment of the present disclosure;

(5) FIG. 4 is a schematic illustration for explaining how some color values obtained within initial images acquired by a camera array are used to compute some interpolated color values, according to an embodiment of the present disclosure;

(6) FIGS. 5a and 5b are example of final images respectively rendered using a classical prior art synthetic aperture refocusing technique (FIG. 5a), and using a proposed technique according to one embodiment of the present disclosure (FIG. 5b), from the same initial images acquired by a four-by-four camera array;

(7) FIGS. 6a and 6b show how depth information may be used to adapt the number of interpolated color values to compute, according to an embodiment of the present disclosure;

(8) FIGS. 7a and 7b illustrate two different synthetic aperture shapes that may be obtained according to an embodiment of the present disclosure, allowing changing the blur shape in a final rendered image;

(9) FIG. 8 is a schematic block diagram illustrating an example of an apparatus for rendering a final image, from at least one image acquired by a camera array, according to an embodiment of the present disclosure.

(10) The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.

5. DETAILED DESCRIPTION

(11) The general principle of the present disclosure relies on a peculiar technique for rendering an image, called a final image, from at least one image acquired by a camera array.

(12) As it will be described more fully hereafter with reference to the accompanying figures, it is proposed in one aspect of the present disclosure to use some interpolated color values, in addition to real color values obtained within the initial images acquired by a camera array, to determine the color value of at least one pixel of a final image to be rendered. In at least one embodiment, the disclosure aims at rendering refocused images in which the angular aliasing in out-of-focused regions is reduced, compared to the angular aliasing that is visible in out-of-focused regions of images rendered with prior art synthetic aperture refocusing technique performed on initial images acquired by a camera array. In other words, one objective of the proposed technique is to provide a blur of better quality in out-of-focus regions of refocused images rendered from initial images acquired by a camera array.

(13) This disclosure may, however, be embodied in many alternate forms and should not be construed as limited to the embodiments set forth herein. Accordingly, while the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the claims. Like numbers refer to like elements throughout the description of the figures.

(14) While not explicitly described, the present embodiments and variants may be employed in any combination or sub-combination.

(15) Some aspects of the general principle of prior art synthetic aperture refocusing technique are now briefly presented, for a better understanding of the various aspects of the present disclosure that will be introduced afterwards. As already mentioned in relation with prior art, synthetic aperture refocusing is a well-known technique that may be used to render a final image from initial images acquired by a camera array. It allows approximating a camera with a large aperture, by aligning, shifting and averaging the initial images acquired by a camera array. By varying the shift, it is possible to focus this synthetic camera at different depths. Synthetic aperture refocusing may thus be used to generate a focal stack from initial images acquired by a camera array. A focal stack is a collection of images of the scene, each of them being focused at a different focalization distance. Images of the focal stack are sometimes referred to as slices of the focal stack. From a focal stack, it is in particular possible to generate extended depth-of-field images, by focus fusion: the in-focus region is detected in each image of a subset of images of the focal stack, then all these in-focus regions are combined to form an extended depth-of-field image.

(16) The process for rendering a slice of focal stack from initial images acquired by a camera array is now briefly explained, at a pixel level. Let assume that the N cameras of a camera array are calibrated, and that we want to render an image S.sup.d, corresponding to the slice of focal stack focused at a focalization distance d from the viewpoint of the i-th camera of the camera array. The image plane of S.sup.d thus coincides with the image plane of the i-th camera of the camera array. The camera array has been calibrated to fit a predetermined model (for example a classical pinhole projection model, or a distorted pinhole projection model). As a result, for each camera of the camera array, a variety of parameters that are well-known in the literaturesuch as camera pose matrix, intrinsic matrix, extrinsic matrix, etc.have already been determined. Using these parameters, it is notably possible to retrieve the corresponding point in the 3D scene at a depth d from the i-th camera of a pixel x belonging to the image plane of the i-th camera. It is then possible to retrieve the pixel x.sub.j.sup.d corresponding to the projection of such a scene point in the j-th camera image plane. In other words, a pixel corresponding to the projection of a 3D scene point can be retrieve in each initial image acquired by a camera array.

(17) The color value of a pixel x of the slice S.sup.d of focal stack rendered at distance d can thus be computed as:

(18) $\begin{matrix} S^{d} (x) = \frac{1}{N} (I_{i} (x) + \underset{j i}{{.Math.}_{j = 1}^{N}} I_{j} (x_{j}^{d})) & (Eq . 1) \end{matrix}$
where I.sub.i(x) represents the color value of the pixel x in the initial image I.sub.i acquired by the i-th camera of the camera array, and I.sub.j(x.sub.j.sup.d) represents the color value of the pixel x.sub.j.sup.d in the initial image acquired by the j-th camera of the camera array.

(19) At the pixel level, the rendering of a slice of focal stack thus consist in retrieving, within each initial images acquired by the camera array, the pixel corresponding to the projection of a given 3D scene point, and then averaging the color values of the N pixels retrieved.

(20) The image slice S.sup.d appears sharp at pixel x when all color values I.sub.1(x.sub.1.sup.d), . . . , I.sub.i(x), . . . , I.sub.N(x.sub.N.sup.d) represent the same color value, meaning that the scene point rendered at pixel x is placed at a depth d from the i-th camera and that this scene point is visible from all cameras points of view, i.e. there's no occlusion.

(21) Conversely, the image slice S.sup.d does not appear sharp at pixel x if I.sub.1(x.sub.1.sup.d), . . . , I.sub.i(x), . . . , I.sub.N(x.sub.N.sup.d) do not all represent a same color value. This is the case when the scene point rendered at pixel x has a depth different than the focalization distance d. This also may be the case if the scene point rendered at pixel x is partially occluded, which means that this scene point is not visible from at least one camera of the camera array (in that case, the subset of cameras that don't see the considered scene point pollute the rendered point: even if the scene point is placed at a depth d from the i-th camera and was therefore supposed to be rendered sharp at pixel x, the color values obtained within said subset of cameras are different than I.sub.i(x), because of occlusions).

(22) From the previous considerations, it should be noticed that the depth of field of a slice of focal stack is very shallow, since only the points of the scene at a distance d from the i-th camera are rendered sharp. Points at a distance d+ or d for a small are already rendered blur.

(23) Considering a three-by-three camera array similar to the one presented in relation with FIG. 1, FIG. 2 is a schematic illustration showing an example of the nine color values I.sub.1(x.sub.1.sup.d), I.sub.2(x.sub.2.sup.d), I.sub.3(x.sub.3.sup.d), I.sub.4(x.sub.4.sup.d), I.sub.5(x), I.sub.6(x.sub.6.sup.d), I.sub.7(x.sub.7.sup.d), I.sub.8(x.sub.8.sup.d), I.sub.9(x.sub.9.sup.d) that may be obtained in association with a pixel x, called a current pixel, of a slice of focal stack S.sup.d to be rendered. In the example of FIG. 2, S.sup.d is the slice of focal stack focused at a distance d from camera C5 of the camera array, and the scene point rendered at pixel x has a depth different than d. As a consequence, the nine color values I.sub.1(x.sub.1.sup.d), . . . , I.sub.4(x.sub.4.sup.d), I.sub.5(x), I.sub.6(x.sub.6.sup.d), . . . , I.sub.9(x.sub.9.sup.d) are not all the same (the different patterns used in FIG. 2 stand for different color values). Current pixel x is thus supposed to be rendered blur in S.sup.d, meaning that it belongs to an out-of-focus region of S.sup.d. According to prior art classical synthetic aperture refocusing technique, only these nine color values are supposed to be averaged to determine the color value of the current pixel x within the slice of focal stack S.sup.d.

(24) As already described in relation with prior art, while the points in S.sup.d appearing sharp are rendered in a pretty good quality, the blur points are poorly rendered as soon as the number of cameras in the camera array is rather small (which is the case with a three-by-three camera array). Indeed, because of the poor sampling of the light-field, this conventional method of Synthetic Aperture Refocusing, when performed on initial images acquired by a camera array, is known to create angular aliasing in the blur regions of the rendered image. FIG. 5a illustrates this drawback of prior art. It shows an example of an image rendered using such a classical synthetic aperture refocusing technique, performed on initial images acquired by a four-by-four camera array. As it can be clearly noticed in this image, blur regions such as region 501 are rendered as the addition of several images where high frequencies are clearly visible: angular aliasing (pattern repetition) appears in these blur regions. The image thus lacks realism and aesthetic quality.

(25) The technique of the present disclosure, which is described below through various embodiments, is aimed at overcoming some of these drawbacks of the prior art. In at least one embodiment, it allows reducing the angular aliasing that may appear within out-of-focus regions of a refocused image rendered from initial images acquired by a camera array.

(26) FIG. 3 is a flow chart for explaining a method for rendering an image, called a final image, from at least one image acquired by a camera array, according to an embodiment of the present disclosure. This method relies on a peculiar technique for determining a color value for at least one pixel of the final image, called a current pixel.

(27) For purposes of simplification, it is assumed that the camera array is a three-by-three camera array similar to the one introduced in relation with FIGS. 1a and 1b, and that the final image to be rendered is a slice of focal stack (being understood that other kind of final images may be rendered, for example by combining different slices of the focal stack rendered with the proposed technique). For example, the final image to be rendered is a slice of focal stack S.sup.d focused at a distance d from the camera C5 of the camera array. The camera array is assumed to be calibrated. As already seen previously, it is thus possible to retrieve, from the current pixel, pixels corresponding to the projection, in each camera (C1, C2, . . . C9) image plane, of the 3D scene point associated with the current pixel at a given depth from camera C5. Throughout the rest of the document, expressions of the form pixel associated with said current pixel within an initial image are used to refer to a pixel of an initial image acquired by a camera array, which corresponds to such a projection of a 3D scene point associated with the current pixel. The notations already used in relation with FIG. 2 are kept (for example, x.sub.j.sup.d represents the pixel corresponding to the projection in the j-th camera image plane, of the 3D scene point associated with the current pixel x at a given depth d from the reference camera C5).

(28) At step 31, for at least one initial image acquired by the camera array, a color value of a pixel associated with said current pixel within said at least one initial image is obtained. Such a color value is referred to as a real color value throughout the rest of the document, in contrast to the notion of interpolated color value that will be introduced afterwards, in relation with step 32.

(29) At step 31, for at least one initial image acquired by the camera array, a color value of at least one pixel, called neighbour pixel, located in the vicinity of the pixel associated with said current pixel within said at least one initial image is obtained. Such a neighbour pixel may be located on a same row or on a same line of the pixel associated with said current pixel. Such a color value is also referred to as a real color value throughout the rest of the document. Step 31 thus acquires a set of real color values for the current pixel and a set of real colors for the at least one neighbour pixel.

(30) At step 32, the real color values previously obtained for the current pixel and the real color values previously obtained for the at least one neighbour pixel are used to compute at least one interpolated color value for the current pixel. The principle of step 32 consists in performing angular interpolation of the color values of the light-field captured by the camera array. Some examples of such an interpolation are presented in relation with FIG. 4. A new color value is for example interpolated for the current pixel from real color values I.sub.1(x.sub.1.sup.d), I.sub.2 (x.sub.2.sup.d), I.sub.4(x.sub.4.sup.d) and I.sub.5(x) obtained for the current pixel at step 31 from initial images I1, I2, I4 and I5 respectively acquired by camera C1, C2, C4 and C5 as well as from real color values I.sub.1(x.sub.1.sup.dk), I.sub.2(x.sub.2.sup.dk), I.sub.5(x.sub.5.sup.dk) and I.sub.4(xk) obtained for the at least one neighbour pixel at step 31 from initial images I1, I2, I4 and I5 respectively acquired by camera C1, C2, C4 and C5. This new color value for the current pixel may be interpolated as (I.sub.1(x.sub.1.sup.d)+I.sub.2(x.sub.2.sup.d)+I.sub.4(x.sub.4.sup.d)+I.sub.5(X)+I.sub.1(x.sub.1.sup.dk)+I.sub.2(x.sub.2.sup.dk)+I.sub.5(x.sub.5.sup.dk)+I.sub.4(xk)), so that such an interpolated color value may be considered as an acceptable approximation of the color value that may have been obtained for the current pixel within an initial image that would have been acquired by a camera placed at the position of the star numbered 1 in FIG. 4, i.e. between cameras C1, C2, C4 and C5 in the camera array. Another interpolation of a color value for the current pixel may be performed from real color values I.sub.8(x.sub.8.sup.d) and I.sub.9(x.sub.9.sup.d) of the current pixel obtained at step 31 from initial images I8 and I9 respectively acquired by camera C8 and C9 as well as from real color values I.sub.8(x.sub.8.sup.dk) and I.sub.9(x.sub.9.sup.dk) of the at least one neighbour pixel obtained at step 31 from initial images I8 and I9 respectively acquired by camera C8 and C9, as (I.sub.8(x.sub.8.sup.d)+I.sub.9(x.sub.9.sup.d)+I.sub.8(x.sub.8.sup.dk)+I.sub.9(x.sub.9.sup.dk)). Again, such an interpolated color value for the current pixel may be considered as an acceptable approximation of the color value that may have been obtained for the current pixel within an initial image that would have been acquired by a camera placed at the position of the star numbered 2 in FIG. 4, i.e. between cameras C8, and C9 in the camera array. In other words, step 32 may be considered, to some extend, as the interpolation of virtual cameras placed between or beside the real cameras of the camera array. Step 32 thus acquires a set of interpolated color values for the current pixel.

(31) At step 33, the color value for the current pixel is determined, as a function of both at least one real color value of a pixel associated to the current pixel as well as a at least one real color value of at least one neighbour pixel. For example, the color value for the current pixel is determined by averaging all the real color values of the pixels associated to the current pixel as well as a at least one real color value of at least one neighbour pixel (in contrast with a classical prior art synthetic aperture refocusing technique, in which only real color values of the current pixel are averaged). In this manner, a camera array comprising more cameras than its number of real cameras may be simulated. For example, a five-by-five camera array may be simulated from a three-by-three camera array when determining the color value of a current pixel, by computing an interpolated color value for every position identified by a star in FIG. 4. The proposed technique thus allows simulating a better angular sampling of the light-field captured by a camera array.

(32) Any interpolation method (nearest neighbours interpolation, bilinear interpolation, etc.) may be applied to compute the interpolated color of of the current pixel. However, according to one embodiment, a nearest neighbors interpolation method is applied, since the inventors have noticed during their test that such an interpolation already produces satisfying results at a small computational cost. It should also be noted that the terms interpolation from at least one real color values used with regard to step 33 should be taken in a broad sense: the computation of a new interpolated color value may be performed not only directly from real color values, but as well from previously interpolated color values (since the real color values are always at the origin of all the interpolated color values).

(33) FIGS. 5a and 5b show examples of final images respectively rendered using a classical prior art synthetic aperture refocusing technique (FIG. 5a), and using a proposed technique according to one embodiment (FIG. 5b). These final images are both rendered from the same initial images acquired by a same camera array (a four-by-four camera array). FIG. 5a has already been described in relation with prior art synthetic aperture refocusing technique. Due to the poor angular sampling of the light-field captured by the camera array, there is angular aliasing and pattern repetition appears in blurred regions such as region 501. FIG. 5b has been rendered according to one embodiment of the rendering technique of a final image. As it can be noticed for example in the out-of-focus region 502, the rendered blur is of better quality when compared to the one rendered in corresponding out-of-focus region 501 of FIG. 5a, with a clear diminution of the angular aliasing.

(34) The general principle of the present technique being now introduced, we present thereafter several embodiments that may be implemented to perform the interpolation of new color values, or to use such interpolated color value when determining the color value of a current pixel of a final image to be rendered. The person skilled in the art will observe that the embodiments depicted can be combined and modified in various forms.

(35) We now assume that the real depth of the scene captured by the camera array is known, and has already been computed with any known method. For example, a depth image (which is sometimes referred to as depth data or depth map in the literature) has already been computed from the initial images acquired by the camera array, and is available. The depth image is for example a 2D representation (i.e. an image) of the 3D scene, in which each pixel is associated with depth or disparity information. Such information is representative of the distance of objects of the scene from a reference point, such as the position of the camera C5, if we still consider a camera array similar to the one introduced in relation with FIGS. 1a and 1b.

(36) According to one embodiment, the rendering method is applied specifically to pixels belonging to an out-of-focus region of the final image. Out-of-focus pixels may easily be identified, since the focalization distance of the final image is known as being predetermined by a user, and since the real depth of the scene is also known (for example through the depth image). No interpolated color values are computed for pixels belonging to a sharp region of the final image (in-focus pixels): as in prior art synthetic aperture refocusing technique, only the real color values are averaged to determine the color value to be rendered for these in-focus pixels. This embodiment allows at the same time improving the blur within out-of-focus region, while saving computational power by avoiding computing unnecessary interpolated color values for pixels appearing sharp within the final image (sharp region of the final image are already rendered in a pretty good qualityabout the same quality as the initial imageswith a classical synthetic aperture refocusing technique).

(37) According to another embodiment of the present disclosure, the number of interpolated color values computed at step 32, and further used at step 33 when determining the color value for a current pixel, is function of a difference between a depth of said current pixel and a depth of an in-focus region of said final image. More particularly, in one embodiment, the number of interpolated color values increases as said difference between a depth of said current pixel and a depth of an in-focus region of said final image increases. Indeed, as already described with reference to prior art, angular aliasing that appears in out-of-focus regions of classical synthetic aperture refocused image becomes even more apparent in out-of-focus regions having high defocus degree with respect to in-focus regions of the rendered image. The proposed embodiment thus allows adapting the number of interpolated color values of the current pixel to that defocused degree. In other words, the higher the defocused degree of an out-of-focus region is, the higher the simulated sampling of the light field is (as already seen, the interpolation of color values may be considered, to some extend, as the interpolation of virtual cameras placed between or beside the real cameras of the camera array). In that way, the computational cost of the implementation of the technique of the present disclosure is optimized, while still preserving good blur quality at every defocus depth. FIGS. 6a and 6b illustrate such an embodiment. On these figures, squares stand for real color values obtained within initial images acquired by the real cameras of a camera array for the current pixel, whereas stars represent interpolated color values for the current pixel (which may be seen as values obtained from initial images acquired by interpolated virtual cameras of the camera array). The configuration of FIG. 6b corresponds to a case in which the considered out-of-focus current pixel has an higher defocus degree than the current pixel considered in the configuration of FIG. 6a, meaning that more virtual cameras are to be rendered in order to have a good angular sampling of the light-field and avoid angular aliasing.

(38) According to one feature of this embodiment, the number of color values to interpolate depends linearly of the defocus degree of the considered current pixel, with respect to a depth d of the in-focus regions of the rendered image. More particularly, the number N.sub.Virtual of virtual cameras interpolated between two real cameras of the camera array for a given current pixel may be defined in several ways. For example N.sub.Virtual may be defined as:
N.sub.Virtual=[[|D(x)d|+]]

(39) where [[.]] is the integer part operator and D(x) is the depth of the current pixel ( and being some predetermined constants).

(40) Alternatively, N.sub.Virtual may be defined as:
N.sub.Virtual=[[s.sub.i.sup.ds.sub.i+1.sup.d]]

(41) where [[s.sub.i.sup.ds.sub.i+1.sup.d]] represents the number of pixels between the projected point s.sub.i.sup.d of the reference camera in the slice image plane and the projected point s.sub.i+1.sup.d of a neighbor camera (such a distance also depends on |D(x)d|).

(42) According to another embodiment, the real color values of the current pixel and the interpolated color values that are used to determine the color value for said current pixel of the final image are determined as a function of a predetermined blur shape. The proposed technique indeed allows defining the shape of a virtual synthetic aperture to be rendered, by interpolating only the specific interpolated color values and by selecting only the specific real color values that will contribute to form that shape. While the configurations described in relation with FIG. 6 illustrate a square aperture that synthetizes the aperture a camera would have if such a camera has a lens of the size of the camera array, other synthetic aperture shapes may be rendered. For example, FIGS. 7a and 7b illustrate two different synthetic aperture shapes that may be created by interpolating more or less specific interpolated color values, and by selecting some specific real color values among the ones that may be obtained. As in FIGS. 6a and 6b, squares stand for real color values for the current pixel obtained within initial images acquired by the real cameras of a camera array, whereas stars represent interpolated color values (which, again, may be seen as values obtained from initial images acquired by interpolated virtual cameras of the camera array). In other words, FIGS. 7a and 7b show two different synthetic aperture shapes that may be created by interpolating more or less virtual cameras and by including all or a subset of the real cameras of a camera array. The fact that the proposed technique enables to create different synthetic aperture shapes is of interest, since the aperture shape has a direct influence on the aesthetic quality of the blur produced in out-of-focus regions of the rendered image. The aesthetic quality of such a blur is usually referred to as the bokeh, which may be defined as the way a lens renders out-of-focus points of light. More particularly, the aperture shape contributes to determine the blur shape within out-of-focus regions of the final image. According to a particular feature, it should be noticed that the present embodiment allows producing different blur shape (or bokeh shape) within different out-of-focus regions of the final image, for example depending on their depths. Indeed, assuming that the depth of a scene is known (though a depth image for example), it is possible to adopt different strategies when computing the interpolated color values from the real color values. For example, the synthetic aperture shape of FIG. 7a may be defined for out-of-focus pixels of a given depth, while the synthetic aperture shape of FIG. 7b may be defined for out-of-focus pixels of a another depth.

(43) According to another embodiment, the determination of the color value of a current pixel, as described in relation with step 33 of FIG. 3, may be done in several ways. It may for example consist in determining an arithmetic average of the real color values of the current pixel and the interpolated color values respectively provided by steps 31 and 32. Alternatively, it may consist in determining a weighted average of said real color values of the current pixel and interpolated color values. For example, a Gaussian kernel may be applied to these values, giving a higher weight to the camera at the centre of a camera array, and a smaller one to the border cameras.

(44) As it can be noticed from the previous described embodiments, the proposed technique not only allows reducing angular aliasing within out-of-focus regions of images rendered from initial images acquired by a camera array, but it also offers many different rendering possibilities that would not be possible with a conventional camera or by performing prior art classical synthetic aperture refocusing technique from initial images acquired by a camera array. More particularly, it is possible to render images with a blur model that can change for every pixel or group of pixels within the rendered image. For example, it is possible to render blur effect that may vary across the image, but may be constant across each same-depth region of the image, usually corresponding to coherent part of the scene (such as identified objects within the scene).

(45) It should also be noticed that the proposed technique is an image post-processing technique: it does not require any material or structural change of the camera array to be implemented. To some extent, it may be applied to a set of initial images of a same scene seen from different points of view that is not acquired by a camera array.

(46) It is to be understood that the present disclosure can be embodied in various forms, and is not to be limited to the examples discussed above. In particular, while the proposed technique has been described mainly with reference to the rendering of final contents that are images, it may also be applied for the rendering of video contents, without departing from the scope of the present disclosure.

(47) FIG. 8 is a schematic block diagram illustrating an example of a device for rendering an image, called a final image, from at least one image acquired by a camera array, according to an embodiment of the present disclosure. Such a device may be any device in which the processing of images acquired by a camera array may be of interest: a computer, a set-top box, a TV set or any portable handheld device such as a smartphone, a tablet, or a camera-array-based light-field capture device for example.

(48) An apparatus 800 illustrated in FIG. 8 includes a processor 801, a storage unit 802, an input device 803, an output device 804, and an interface unit 805 which are connected by a bus 806. Of course, constituent elements of the computer apparatus 800 may be connected by a connection other than a bus connection using the bus 806.

(49) The processor 801 controls operations of the apparatus 800. The storage unit 802 stores at least one program to be executed by the processor 801, and various data, including for example depth information, parameters used by computations performed by the processor 801, intermediate data of computations performed by the processor 801, and so on. The processor 801 is formed by any known and suitable hardware, or software, or a combination of hardware and software. For example, the processor 801 is formed by dedicated hardware such as a processing circuit, or by a programmable processing unit such as a CPU (Central Processing Unit) that executes a program stored in a memory thereof.

(50) The storage unit 802 is formed by any suitable storage or means capable of storing the program, data, or the like in a computer-readable manner. Examples of the storage unit 802 include non-transitory computer-readable storage media such as semiconductor memory devices, and magnetic, optical, or magneto-optical recording media loaded into a read and write unit. The program causes the processor 801 to perform a process for rendering an image, called a final image, from at least one image acquired by a camera array, according to an embodiment of the present disclosure as described previously. More particularly, the program causes the processor 802 to determine a color value to apply to at least one pixel of the final image, called a current pixel. The real color values of the current pixel and interpolated color values used to determine this color value of the current pixel may be stored into storage unit 802.

(51) The input device 803 is formed for example by a keyboard, a pointing device such as a mouse, or the like for use by the user to input commands. In embodiments wherein the final image rendering is performed within a camera-array-based light-field capture device itself, the input device 803 also comprises the optical device used to capture the scene.

(52) The output device 804 is formed for example by a display device to display, for example, the final image determined by applying the method for rendering previously described. The input device 803 and the output device 804 may be formed integrally by a touchscreen panel, for example. The input device 803 may be used by an operator for selecting regions that are to be in focus within the rendered image.

(53) The interface unit 805 provides interfaces between the apparatus 800 and external apparatus. The interface unit 805 may be communicable with external apparatus via cable or wireless communication. In one embodiment, the external apparatus may be a display device, for displaying the final image rendered by the rendering device.

(54) Although only one processor 801 is shown on FIG. 8, it must be understood that such a processor may comprise different modules and units embodying the functions carried out by apparatus 800 according to embodiments of the present disclosure, such as modules allowing determining a color value for at least one pixel of the final image, called a current pixel: a module for obtaining, for at least one initial image acquired by a camera array, a color value of a pixel associated with said current pixel within said at least one initial image, acquiring at least one color value called a real color value; a module for obtaining, for at least one initial image acquired by a camera array, a color value of a neighbour pixel, acquiring at least one color value called a real color value; a module for obtaining at least one interpolated color value from the previously acquired real color values; a module for determining the color value for said current pixel, as a function of said at least one real color value of an associated pixel and said at least one interpolated color value.

(55) These modules and units may also be embodied in several processors 801 communicating and co-operating with each other.

Method for rendering a final image from initial images acquired by a camera array, corresponding device, computer program product and computer-readable carrier medium

Assignee

Inventors

Cpc classification

Classification Explorer

G06T5/50

PHYSICS

Classification Explorer

H04N2013/0088

ELECTRICITY

Classification Explorer

G06T7/557

PHYSICS

Classification Explorer

H04N23/13

ELECTRICITY

Classification Explorer

G06T2207/10024

PHYSICS

Classification Explorer

G06T5/002

PHYSICS

Classification Explorer

G06T2207/10052

PHYSICS

Classification Explorer

G06T7/90

PHYSICS

Classification Explorer

H04N23/90

ELECTRICITY

International classification

Classification Explorer

H04N5/225

ELECTRICITY

Classification Explorer

G06T5/00

PHYSICS

Classification Explorer

G06T7/90

PHYSICS

Classification Explorer

G06T7/557

PHYSICS

Classification Explorer

H04N5/247

ELECTRICITY

Classification Explorer

H04N9/09

ELECTRICITY

Classification Explorer

G06T5/50

PHYSICS

Abstract

Claims

Description