Method for processing a light field video based on the use of a super-rays representation

Abstract

A method and device for processing a light field video is described. The light field video includes a set of image views per unit of time, the light field video being associated with a scene without cuts. In the method a first super-rays representation of reference image views at a given time is determined based on centroids. A second super-rays representation associated with corresponding views of a subsequent set of image views is next determined based on de-projection and re-projection of centroids. The displacement of centroids between the first and second super-rays is determined and then the determined displacement is applied to centroids of the second super-rays representation.

Claims

1. A method comprising: obtaining a light field video comprising a set of image views per unit of time, the light field video being associated with a scene without cuts; determining a first super-rays representation of at least one reference image view being comprised in said set of image views at a given time, said super-rays representation being based on centroids, and each centroid being associated with a unique super-ray; determining a second super-rays representation of at least one corresponding reference view in a subsequent set of image views following said given time, based on a de-projection and re-projection of centroids of said at least one reference image view; determining a displacement of centroids between said first and second super-rays representation based on a tracking process; and applying a determined displacement on centroids of said second super-rays representation for obtaining positions of modified centroids.

2. The method for processing according to claim 1, wherein it further comprises de-projecting and re-projecting of modified centroids onto at least one another image view in said subsequent set of image views, for obtaining a super-rays representation of said subsequent set of image views.

3. The method for processing according to claim 2, wherein it further comprises updating the position of said centroids by taking into account an average of values of projected rays, said rays being in a neighborhood of centroids in said first super-rays representation.

4. The method for processing according to claim 1, wherein said tracking process is a minimization process that comprises the determination of argmin.sub.δ.sub.c.sub.x.sub.,δ.sub.c.sub.y.sub.,δ.sub.c.sub.zΣ.sub.s,t∥RGB.sup.f(P.sub.s,t.sup.d.sup.c(x.sub.c.sup.f,y.sub.c.sup.f))−RGB.sup.f+1(P.sub.s,t.sup.d.sup.c.sup.+δ.sup.c.sup.d(x.sub.c.sup.f+δ.sub.c.sup.x,y.sub.c.sup.f+δ.sub.c.sup.y))∥.sup.B, where RGB.sup.f(P.sub.s,t.sup.d(x,y)) corresponds to the RGB pixel value of the projection of a ray r of coordinates (s, t, x, y) at a depth d, associated with a frame f.

5. The method for processing according to claim 1, wherein at least two reference images views are used, and said at least two reference image views are far from each other's from an angular point of view.

6. The method for processing according to claim 1, wherein said light field video has been acquired by a cameras array.

7. The method for processing according to claim 1, wherein said light field video has been acquired by a plenoptic camera.

8. An electronic device comprising at least one processor, and a memory unit coupled to said at least one processor, the at least one processor being configured to: obtain a light field video comprising a set of image views per unit of time, the light field video being associated with a scene without cuts; determine a first super-rays representation of at least one reference image view being comprised in said set of image views at a given time, said super-rays representation being based on centroids, and each centroid being associated with a unique super-ray; determine a second super-rays representation of at least one corresponding reference view in a subsequent set of image views following said given time, based on a de-projection and re-projection of centroids of said at least one reference image view; determine a displacement of centroids between said first and second super-rays representation based on a tracking process; and apply a determined displacement on centroids of said second super-rays representation for obtaining positions of modified centroids.

9. The electronic device for processing according to claim 8, wherein the at least one processor is further configured to de-project and re-project modified centroids onto at least one another image view in said subsequent set of image views, for obtaining a super-rays representation of said subsequent set of image views.

10. The electronic device for processing according to claim 9, wherein the at least one processor is further configured to update the position of said centroids by taking into account an average of values of projected rays, said rays being in a neighborhood of centroids in said first super-rays representation.

11. The electronic device for processing according to claim 8, wherein said tracking process is a minimization process that comprises the determination of argmin.sub.δ.sub.c.sub.x.sub.,δ.sub.c.sub.y.sub.,δ.sub.c.sub.zΣ.sub.s,t∥RGB.sup.f(P.sub.s,t.sup.d.sup.c(x.sub.c.sup.f,y.sub.c.sup.f))−RGB.sup.f+1(P.sub.s,t.sup.d.sup.c.sup.+δ.sup.c.sup.d(x.sub.c.sup.f+δ.sub.c.sup.x,y.sub.c.sup.f+δ.sub.c.sup.y))∥.sup.B, where RGB.sup.f(P.sub.s,t.sup.d(x,y)) corresponds to the RGB pixel value of the projection of a ray r of coordinates (s, t, x, y) at a depth d, associated with a frame f.

12. The electronic device for processing according to claim 8, wherein at least two reference images views are used, and said at least two reference image views are far from each other's from an angular point of view.

13. The electronic device for processing according to any claim 8, wherein said light field video has been acquired by a cameras array.

14. The electronic device for processing according to any claim 8, wherein said light field video has been acquired by a plenoptic camera.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The above and other aspects of the invention will become more apparent by the following detailed description of exemplary embodiments thereof with reference to the attached drawings in which:

(2) FIG. 1 presents a flowchart corresponding to the main steps that are executed for determining a super-rays representation for a given frame;

(3) FIG. 2 presents the flowchart of FIG. 1 as an algorithm;

(4) FIG. 3(a) presents a light field image corresponding to a set of 9 images;

(5) FIG. 3(b) presents the image view which is the upper-left image view in the set of images views presented in FIG. 3(a);

(6) FIG. 3(c) presents a super-rays representation of the image view presented in FIG. 3(b), when the process of FIG. 1 or 2 is applied on the light field image of the FIG. 3(a);

(7) FIG. 4(a) presents in a schematic way a method for determining a super-rays representation of a light field video;

(8) FIG. 4(b) presents an illustration of the execution of the method for determining a super-rays representation of a light field video according to one embodiment of the disclosure;

(9) FIG. 5(a) presents a search zone for an approximate neighborhood used in the method for determining a super-rays representation of a light field video, according to one embodiment of the disclosure;

(10) FIG. 5(b) presents the steps of the method for determining a super-rays representation of a light field video, described as an algorithm, in one embodiment of the disclosure;

(11) FIG. 6 presents an example of a device that can execute a process according to one embodiment of the disclosure.

DETAILED DESCRIPTION

(12) FIG. 1 presents a flowchart corresponding to the main steps that are executed for determining a super-rays representation for a given frame (being a set of image views). Such given frame belongs to a sequence of a light field video, and the given frame is considered at the first frame (or close to the first frame) in such sequence. The other frames in the sequence are going to be processed differently for obtaining a super-rays representation.

(13) The purpose of processing described in FIG. 1 is to determine a super-rays representation of the image views comprised in the given frame. For reminders, a super-ray is described by its centroid ray r.sub.c. Therefore, by definition, all the light-rays in a Super-Ray of centroid r, are labeled c. Now, a centroid light ray r.sub.c at the given frame f.sub.0 is represented by four coordinates (the reference view s.sub.0, t.sub.0 coordinates and spatial coordinates x.sub.c.sup.f.sup.0,y.sub.c.sup.f.sup.0) and a disparity value d.sub.c.sup.f.sup.0. Each centroid ray is initialized with a color Lab.sub.c.sup.f.sup.0 (initialised with the reference view color). Let us explain more precisely how the super-rays are estimated on the given frame:

(14) In a step referenced 101, an electronic device performs an initialization step. More precisely, in one embodiment of the disclosure, it is proposed to use the central view as a reference and seed the spatial coordinates of the centroids on a regular grid of step S. The sparse depth for each seed is supposed to be known or it can be computed with any well-known method of the state-of-the-art.

(15) Then, in a step referenced 102, the electronic device performs an assignment step. More precisely, the electronic device determines the ray centroid projection on each image view, using the estimated depth d.sub.c, and assign the same label of the centroid ray c to the pixels in a small window of the projection pixel having similar color and small distance.

(16) Formally, we denote P.sub.s′,t′.sup.d(r) the spatial coordinates of the projection of a ray r of coordinates

(17) (s.sub.r,t.sub.r,y.sub.r.sup.f.sup.0,y.sub.r.sup.f.sup.0) and depth d to another view (s′, t′):
P.sub.s′,t′.sup.d(r)=(d(s.sub.r−s′)+x.sub.r.sup.f.sup.0,d(t.sub.r−t′)+y.sub.r.sup.f.sup.0).
It should be noted that the equation above stands for a light-field with rectified views but generalize to the case of unrectified views using the calibration parameters.
If Δ.sub.Lab and Δ.sub.xy denote respectively the Lab color difference and the spatial Euclidean distance, N.sub.S×S denote the S×S window around a pixel and m a parameter balancing importance of color versus spatial compactness, then the assignment step 102 can be summarized as the determination of a label for each ray r, that is associated with a centroid (i.e. rays are grouped according to centroids):

(18) $A\left(r\right)=\underset{c}{\mathrm{argmin}}\left\{{\Delta }_{\mathrm{Lab}}\left(r_c,r\right)+m.Math.{\Delta }_{\mathrm{xy}}\left(P_{s_r,t_r}^{d_c}\left(r_c\right),r\right).Math.r\in N_{S\times S}\left(P_{s_r,t_r}^{d_c}\left(r_c\right)\right)\right\}$

(19) Doing so, we establish a correspondence, not only for light rays similar in appearance in one view, but also for light rays captured in all the views that are similar in appearance and likely to come from the same scene point. It should be noted that the S×S window around a pixel can range from 10×10 pixels to 100×100 pixels depending on the image resolution.

(20) Then, in a step referenced 103, the electronic device performs an update step. More precisely, the new color of each super-ray is determined by averaging the color values of all light-rays having the same label. The new position of the centroid in the reference view is obtained averaging the projected coordinates on the reference view of all light-rays having the same label:
Lab.sub.c=mean(Lab.sub.r), xy.sub.c=mean(P.sub.s.sub.0.sub.,t.sub.0.sup.d.sup.c(r))
∀r|A(r)=c.
When the centroid positions are not changed or modified, or a maximum number of iterations is reached, the iterative process stops. In one embodiment of the disclosure, it is proposed to use a number of iterations equal to 10.

(21) In addition, in a facultative step referenced 104, the electronic device can perform a cleanup step. Indeed, similarly to the SLIC method, the execution of steps 101 to 103 does not enforce super-ray spatial connectivity. Therefore, after the grouping of rays around centroids, some rays may remain isolated, especially when the spatial term in the equation

(22) $A\left(r\right)=\underset{c}{\mathrm{argmin}}\left\{{\Delta }_{\mathrm{Lab}}\left(r_c,r\right)+m.Math.{\Delta }_{\mathrm{xy}}\left(P_{s_r,t_r}^{d_c}\left(r_c\right),r\right).Math.r\in N_{S\times S}\left(P_{s_r,t_r}^{d_c}\left(r_c\right)\right)\right\}$
has a low weight. Hence, the electronic device can perform in step 104 a simple post-processing method that comprises the re-labeling of super-ray disconnected components (with a number of pixels below a value of ¼S.sup.2) with the closest super-ray label.

(23) FIG. 2 presents the flowchart of FIG. 1 as an algorithm.

(24) FIG. 3(a) presents a light field image corresponding to a set of 9 images.

(25) FIG. 3(b) presents the image view which is the upper-left image view in the set of images views presented in FIG. 3(a).

(26) FIG. 3(c) presents a super-rays representation of the image view presented in FIG. 3(b), when the process of FIG. 1 or 2 is applied on the light field image of the FIG. 3(a).

(27) It should be noted that some “area” associated with a super-rays seem to have more than one centroid (see for example on the lamp). However, the centroids presented in FIG. 3(c) corresponds to the superposition of all the centroids from all the image views of FIG. 3(a) (i.e. a kind of map of all the determined centroids from all the image views).

(28) It should be noted that super-rays representation can be obtained via other technique. Indeed, in a broader interpretation, it is proposed, in another embodiment of the disclosure, to assimilate or define a super-rays representation of a light-field image as a segmented representation of a light-field image obtained via other techniques consisting in gathering rays from the light field image into groups of rays, or in assigning labels associated with groups of rays from the light field image. Then, centroids (or geometric centers) are determined once the segmentation of the light field image has been done.

(29) Indeed, it is necessary to define centroids in order to implement the technique described in FIGS. 4 to 5. For example, by applying the technique described in the article entitled “4D Light Field SuperPixel and Segmentation” by Zhu et al. published in Computer Vision and Pattern Recognition (CVPR) 2017, it is possible to obtain Light Field super-pixels with an EPI representation. Then, after the execution of a conversion process for obtaining the resulting segmentation of a light-field image represented as a matrix of image views, the centroids are determined.

(30) In a variant, other techniques used for determining super-pixels can be adapted for obtaining super-rays representation. Indeed, it is possible to modify these techniques by taking into account the strong correlation between the image views. The output of these adapted techniques is still named as a super-rays representation in the following.

(31) In the following, we focus on the determination of a super-rays representation of a sequence of frames within a light field video. For example, the sequence of frames comprises N frames to be processed, and each frame being associated with a matrix of image views.

(32) In one embodiment of the disclosure, an electronic device performs for the first frame f the determination of a super-rays representation based on the process depicted in FIGS. 1 and 2. Then, for a subsequent frame f+1, according to one embodiment of the disclosure, it is proposed to take into account the movement of objects in the scene for determining the super-rays representation of the subsequent frame.

(33) In one embodiment of the disclosure, the electronic device determines the displacement in x, y and d, noted δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z, of each centroid ray, for a reference view between frames f and f+1. This problem is known as scene flow, or in our case sparse scene flow.

(34) The electronic device can solve such problem by searching for the vector δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z that minimizes the color distance in a patch around each centroid: (δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z)=argmin.sub.δ.sub.c.sub.x.sub.,δ.sub.c.sub.y.sub.,δ.sub.c.sub.zΣ.sub.s,t∥RGB.sup.f(P.sub.s,t.sup.d.sup.c(x.sub.c.sup.f,y.sub.c.sup.f))−RGB.sup.f+1(P.sub.s,t.sup.d.sup.c.sup.+δ.sup.c.sup.d(x.sub.c.sup.f+δ.sub.c.sup.x,y.sub.c.sup.f+δ.sub.c.sup.y))∥.sup.B where RGB.sup.f is the color value at a frame f and ∥.sup.B denotes the sum of squared difference in a patch of size B. Hence, according to such approach, the centroids of all the images views in the frame f+1 are determined by applying to all of the centroids of all image views in the frame f, the corresponding set of displacement vectors (i.e. the vectors δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z associated with a given centroid) previously determined.

(35) In another embodiment of the disclosure, it is proposed to split the search into a x, y displacement search on only a reference image view, along with an update of the depth value. The reference view can be the central image view or another chosen image view.

(36) In another embodiment, δ.sub.c.sup.z can be determined using a block-matching technique, and δ.sub.c.sup.x, δ.sub.c.sup.y can be determined computed using Deep Match as described in the document entitled “Deepmatching: Hierarchical deformable dense matching” by Revaud et al. published in International Journal of Computer Vision 120(3): 300-323, 201t.

(37) FIG. 4(a) presents in a schematic way a method for determining a super-rays representation of a light field video. The frame f that comprises 4 image views of a scene, at a given time (t=0). After having applied the method or process depicted in FIG. 1 or FIG. 2 on the frame f, a super-rays representation is obtained, and each super-ray is associated with a centroid (the centroids referenced 401 in the FIG. 4). Then, a reference image view is selected in the frame f. The reference image view can be a central image view in one embodiment of the disclosure. Then, the centroids of the reference image are projected on the corresponding image view in a subsequent frame (the frame f+1, for a time t=1), and a vector δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z is determined according to the previous described method. For example, the centroid with coordinates (x.sub.c.sup.f, y.sub.c.sup.f) in the reference image view is projected in the corresponding image view in the frame f+1, and the displacement vector δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z is determined, enabling to determine a modified centroid 402. Each centroid of the reference image view is associated with its own displacement vector δ.sub.c.sup.x,δ.sub.c.sup.y,δ.sub.c.sup.z.

(38) Then, in one embodiment of the disclosure, once the modified centroids 402 within the equivalent of the reference image view in the frame f+1 are obtained, the electronic device performs a de-projection and re-projection method (as accurate values of the modified centroids 402 in depth are obtained from the value z.sub.c.sup.f and δ.sub.c.sup.z) onto the other image views of the frame f+1. In one embodiment of the disclosure, the projection/deprojection function can be the function P.sub.s′,t′.sup.d(r) previously mentioned. For an arbitrary camera-array geometry, the standard epipolar geometry relationship (described in the chapter 11 of the document entitled “Computer Vision: Algorithms and Applications” by Richard Szeliski) can be used for performing the de-projection and re-projection method.

(39) In a variant, instead of using only one reference image view between the frame f and f+1, it is proposed to use several reference image views. Such embodiment is of interest when occlusions in a scene occurs. It should be noted that in the case that, for a given image view, centroids coming from the re-projection/de-projection process from centroids of reference image views of apparently a same super-rays do not exactly coincide, it is proposed to define a unique centroid as being positioned at the middle of each segment that links the coordinates (according to x, y or z axis) of these close centroids.

(40) In one embodiment of the disclosure, it is proposed to use two reference images views that are far from each other (in term of angular distance) within the matrix or set of image views.

(41) It should be noted that once the centroids at frame f+1 have been initialized or determined via the previous mentioned technique, in one embodiment of the disclosure, the electronic device performs an update step 103. In a variant, the electronic device also performs a clean-up stage 104 on the image views of the frame f+1.

(42) FIG. 4(b) presents an illustration of the execution of the method for determining a super-rays representation of a light field video according to one embodiment of the disclosure.

(43) More precisely, the foreground super-ray referenced 403 is tracked over the consecutive frames of a 2×1 light field images. Other super-rays do not move since the background is static. The depth d is used to enforce angular consistency from a view to another, while the scene flow δ.sub.x, δ.sub.y guarantees temporal consistency. On Frame 3, the moving super-ray 403 becomes too close of the super-ray referenced 406, and too far from the super-ray referenced 404, triggering the creation of the super-ray referenced 405, and the deletion of the super-ray referenced 406.

(44) It should be noted that in order to cope with the low-frame rates of the current light field camera array, giving large object displacement on the sensor, a feature matching technique as described in the article entitled “DeepFlow: Large displacement optical flow with deep matching” by Weinzaepfel, Philippe, et al, published in the proceedings of the IEEE International Conference on Computer Vision, 2013, can be used. Such approach can be efficiently implemented on a GPU processor. A list a correspondence is obtained at a scale N (in our case N=3) from on the views s.sub.0 t.sub.0 between f and f+1. With such approach, the electronic device can obtain a set M of matches m with coordinates [(x.sub.m.sup.f,y.sub.m.sup.f),(x.sub.m.sup.f+1,y.sub.m.sup.f+1)]. Then, in order to determine the value for each centroid and avoid noisy or wrong matches, a trilateral filter using color, spatial and matching confidence to assign a displacement value for each super-ray centroid can be used:

(45) $\left({\delta }_c^x,{\delta }_c^y\right)=\frac{1}{W}\underset{m\in M}{.Math.}\exp \left(-\frac{{.Math.\begin{array}{c}\left(x_m^f,y_m^f\right),\\ \left(c_x^f,c_y^f\right)\end{array}.Math.}^{⩓2}}{{\sigma }_{\mathrm{xy}}}-\frac{{.Math.\begin{array}{c}{\mathrm{Lab}}^f\left(x_m^f,y_m^f\right),\\ {\mathrm{Lab}}^f\left(c_x^f,c_y^f\right)\end{array}.Math.}^{⩓2}}{{\sigma }_{\mathrm{Lab}}}-\frac{c_m}{{\sigma }_m}\right).\left(x_m^{f+1}-x_m^f\right)$$\mathrm{With}$$W=\underset{m\in M}{.Math.}\exp \left(-\frac{{.Math.\begin{array}{c}\left(x_m^f,y_m^f\right),\\ \left(c_x^f,c_y^f\right)\end{array}.Math.}^{⩓2}}{{\sigma }_{\mathrm{xy}}}-\frac{{.Math.\begin{array}{c}{\mathrm{Lab}}^f\left(x_m^f,y_m^f\right),\\ {\mathrm{Lab}}^f\left(c_x^f,c_y^f\right)\end{array}.Math.}^{⩓2}}{{\sigma }_{\mathrm{Lab}}}-\frac{c_m}{{\sigma }_m}\right),$
C.sub.m the confidence value of a match and σ.sub.xy, σ.sub.Lab, σ.sub.m tree parameters to control the importance of each term. It should be noted that the depth is updated by searching in a small window the depth value that minimizes the color distance
δ.sub.c.sup.z=argmin.sub.δ.sub.c.sub.zΣ.sub.s,t∥RGB.sup.f(P.sub.s,t.sup.d.sup.c(x.sub.c.sup.f,y.sub.c.sup.f))−RGB.sup.f+1(P.sub.s,t.sup.d.sup.c.sup.+δ.sup.c.sup.d(x.sub.c.sup.f+δ.sub.c.sup.x,y.sub.c.sup.f+δ.sub.c.sup.y))∥.sup.B

(46) Then, once each centroid is displaced on a subsequent frame, a new round of assignment and update is run as proposed in the article entitled “Temporally consistent superpixels” by Reso, Matthias, et al., published in the proceedings of the IEEE International Conference on Computer Vision. 2013.

(47) However, the technique of Reso et al. is interested in object segmentation and not so much in having superpixels that are temporally consistent. If we want the super rays to be consistent form a frame to another, we must prevent the centroids to move too much from their position after the update. In other words, super-rays belonging to static parts of the scene should not move in time. To do so we run 5 iterations of the assignment step 102 followed by a modified update step. The modified update step uses the position of the centroid after the move with the flow to limit the centroid displacement:
(x.sub.m.sup.f+1,y.sub.m.sup.f+1)=(1−p).Math.mean(P.sub.s.sub.0.sub.,t.sub.0.sup.d.sup.c(r))+p.Math.(x.sub.c.sup.f+δ.sub.c.sup.x,y.sub.c.sup.f+δ.sub.c.sup.y)
∀r|A(r)=c.

(48) The parameter p allows to control how much the previous centroids position matter in the computation of the new one. In the case when p=0, the update step 103, when p=1, the centroids are prohibited from moving after the flow propagation.

(49) In a variant, it is proposed to determine the following:

(50) $\left(x_c^{f+1},y_c^{f+1}\right)=p^{\prime }.Math.\frac{1}{.Math.{\mathrm{SR}}_c^{f+1}.Math.}\underset{r\in {\mathrm{SR}}_c^{f+1}}{.Math.}\left(P_{s_c}^{d_c}\left(x_r^f\right),P_{t_c}^{d_c}\left(y_r^f\right)\right)+\left(1-p^{\prime }\right).Math.\left(x_c^f+{\delta }_c^x,y_c^f+{\delta }_c^y\right)$

(51) where p′ is also a parameter that controls how much the super-rays are allowed to move from their theoretical position. When p′=1, this step corresponds to the same SLIC iteration. Newly created centroids from the step described previously always have p′=1, allowing them to adapt to the scene changes. When p′=0, the super-rays centroids are not allowed to move at all, providing the best consistency.

(52) Because occlusion, deformable objects, or object entering or leaving the scene, the tracking of a super-ray can be lost from a frame to another. To cope with this, we leverage the depth information provided by our super-rays with the following strategy.

(53) We begin to compute an approximate right, left, up and down neighborhood for each centroid, as illustrated in FIG. 5(a). Indeed, FIG. 5(a) presents a search zone for our approximate neighborhood in which the selected centroid is referenced 501 and potential neighborhoods are referenced 502. The dark zone (with a cross shape) referenced 503 illustrates the cardinal points search area M.

(54) Let M.sup.up(c.sub.1), M.sup.down(c.sub.1), M.sup.left(c.sub.1), M.sup.right(c.sub.1) be the list of centroids that lies around the 4 cardinal axis of a centroid of index c.sub.1, e.g
M.sup.left(c.sub.1)={c.sub.2|x.sub.c.sub.2<x.sub.c.sub.1 and |y.sub.c.sub.2−y.sub.c.sub.1|<S}

(55) Where S the size of a superpixel.

(56) The approximate neighborhood of a centroid is N(c)={N.sup.up(c.sub.1), N.sup.down(c.sub.1), N.sup.left(c.sub.1), N.sup.right(c.sub.1)} with
N.sup.left(c.sub.1)=argmin.sub.c.sub.2.sub.∈M.sub.up|y.sub.c.sub.1−y.sub.c.sub.2|

(57) To decide which centroid to update, we use the following new rules based of the spatial and color distance and the depth:

(58) If a centroid becomes to close from it neighborhood, the one centroid which is ‘behind’, i.e. with the largest assigned depth d.sup.c it is placed on a list of centroids to delete.

(59) If a centroid is too far from one of its neighborhood, both centroids are placed on the list of centroids that will be used for creating a new centroid at the middle of the two centroids.

(60) At the end of the search, a number k equal to the greatest number of element in either of the list of centroids are deleted and created.

(61) The depth of the created super-rays is recomputed from scratch or using the depth value of its neighborhood;

(62) The centroid that have been created are updated using p=0 to allow them to move to adapt to potential contours.

(63) The approach has been tested on synthetic and real data. We saw that the centroids are in fact consistent from a frame to another, and still consistent from a view to another.

(64) Because super-rays offer a temporally and angularly consistent representation of the light fields, it is easy to imagine how it can be used to reduce the size of a light field video, in the context of image compression.

(65) For instance, in the article entitled “Superpixel-driven graph transform for image compression”, by Fracastoro, Giulia, et al, published in the Image Processing (ICIP), 2015 IEEE International Conference on. IEEE, 2015. it is proposed a technique that uses graph-based Fourier transform computed for clusters of super pixel to encode a single image. The encoded segments and the coefficient are then transmitted, outperforming the DCT both in term of quality and bitrate. Is it not hard to image this gain multiplies if a single transform can be used for segments between views and frames.

(66) Another possible application leverages the depth of each centroid. Indeed, supposing we want to insert an object of a light field into another light field (video or static), segmenting the object as in the article entitled “Super-rays for Efficient Light Field Processing”. The depth of the centroid can be used as a z-buffer to decide which pixel to render on each view. Additionally, if color or style transfer needs to be run to homogenize the two light fields, super-rays can be used to guaranty the angular and temporal consistency of the transfer (for example, in the article entitled “Superpixel-based Color Transfer” by Giraud, Remi, Vinh-Thong Ta, and Nicolas Papadakis, published in IEEE International Conference on Image Processing (ICIP), 2017, it is proposed a super-pixel based color transfer, we can imagine having a single color transform by dynamic (or static) super ray to avoid temporal or angular flickering).

(67) FIG. 5(b) presents the steps of the method for determining a super-rays representation of a light field video, described as an algorithm, in one embodiment of the disclosure.

(68) Indeed, it is proposed to process the first frame of a sequence of a light field frames (being a part of a light field video) as in the article entitled “Super-rays for Efficient Light Field Processing”.

(69) Then, the centroids are moved by using determined movement vectors δ.sub.c.sup.x,δ.sub.c.sup.y, δ.sub.c.sup.z.

(70) Then, because of object movements in the scene, parts of the super-rays can be occluded and disoccluded or enter and leave the light field. For this reason, creating or deleting super-rays might be necessary. While the superpixel size or color consistency has been used to determine the creation or deletion in other research works, we propose to leverage the depth information associated to the super-ray to detect occlusions and disocclusions.

(71) At last, to take into account non-rigid deformation and super-ray partial occlusion, the segmentation needs to be updated to fit the new frame.

(72) In the article “Temporally consistent superpixels” by M. Reso, J. Jachalsky, B. Rosenhahn, and J. Ostermann, published in Proceedings of the IEEE International Conference on Computer Vision, pages 385-392, 2013, five SLIC iterations are run, where the centroids are allowed to move freely. This has for consequence that superpixels of static objects, that should not change are affected by the creation, deletion and movements of nearby superpixels. To compensate for this, we enforce the super-pixel centroids to take into account its position prior to the k-means iteration.

(73) FIG. 6 presents an example of an electronic device that can be used to perform one or several steps of methods disclosed in the present document.

(74) Such electronic device referenced 600 comprises a computing unit (for example a CPU, for “Central Processing Unit”), referenced 601, and one or more memory units (for example a RAM (for “Random Access Memory”) block in which intermediate results can be stored temporarily during the execution of instructions a computer program, or a ROM block in which, among other things, computer programs are stored, or an EEPROM (“Electrically-Erasable Programmable Read-Only Memory”) block, or a flash block) referenced 602. Computer programs are made of instructions that can be executed by the computing unit. Such electronic device 600 can also comprise a dedicated unit, referenced 603, constituting an input-output interface to allow the device 600 to communicate with other electronic devices. In particular, this dedicated unit 603 can be connected with an antenna (in order to perform communication without contacts), or with serial ports (to carry communications “contact”). It should be noted that the arrows in FIG. 6 signify that the linked unit can exchange data through buses for example together.

(75) In an alternative embodiment, some or all of the steps of the method previously described, can be implemented in hardware in a programmable FPGA (“Field Programmable Gate Array”) component or ASIC (“Application-Specific Integrated Circuit”) component.

(76) In an alternative embodiment, some or all of the steps of the method previously described, can be executed on an electronic device comprising memory units and processing units as the one disclosed in the FIG. 6.

(77) In one embodiment of the disclosure, the electronic device depicted in FIG. 6 can be comprised in a camera device that is configured to capture images (either conventional 2D images or a sampling of a light field). These images are stored on one or more memory units. Hence, these images can be viewed as bit stream data (i.e. a sequence of bits). Obviously, a bit stream can also be converted on byte stream and vice versa.