Methods and apparatuses for encoding and decoding digital light field images

Abstract

A method for encoding a raw lenselet image includes a receiving phase, wherein at least a portion of a raw lenselet image is received, the image including a plurality of macro-pixels, each macro-pixel having pixels corresponding to a specific view angle for the same point of a scene, and an output phase, wherein a bitstream having at least a portion of an encoded lenselet image is outputted. The method has an image transform phase, wherein the pixels of said raw lenselet image are spatially displaced in a transformed multi-color image having a larger number of columns and rows with respect to the received raw lenselet image, wherein dummy pixels having undefined value are inserted into the raw lenselet image and wherein the displacement is performed so as to put the estimated center location of each macro-pixel onto integer pixel locations. Moreover, the method includes a sub-view generation phase, wherein a sequence of sub-views is generated, said sub-views having pixels of the same angular coordinates extracted from different macro-pixels of the transformed raw lenselet image. Finally, the method has a graph coding phase, wherein a bitstream is generated by encoding a graph representation of at least one of the sub-views of the sequence according to a predefined graph signal processing technique.

Claims

1. A method for encoding a raw lenselet image (f), comprising: a receiving phase, wherein at least a portion of a raw lenselet image (f) is received, said image (f) comprising a plurality of macro-pixels, each macro-pixel comprising pixels corresponding to a specific view angle for the same point of a scene; an output phase, wherein a bitstream (f.sub.d{circumflex over ()}) comprising at least a portion of an encoded lenselet image (f) is outputted; an image transform phase, wherein the pixels of said raw lenselet image (f) are spatially displaced in a transformed multi-color image that includes the colors red, blue and green and has a larger number of columns and rows with respect to the received raw lenselet image, wherein dummy pixels having undefined value are inserted into said raw lenselet image (f) and wherein said displacement is performed so as to put the estimated center location of each macro-pixel onto integer pixel locations; a sub-view generation phase, wherein a sequence of sub-views (f.sub.d) is generated, said sub-views comprising pixels of the same angular coordinates extracted from different macro-pixels of said transformed raw lenselet image (f); a graph coding phase, wherein a bitstream (f.sub.d{circumflex over ()}) is generated by encoding a graph representation of at least one of the sub-views of said sequence (f.sub.d) according to a predefined graph signal processing (GSP) technique, wherein said output phase comprises outputting said graph-coded bitstream (f.sub.d.sup.) for its transmission or storage.

2. The encoding method according to claim 1, wherein said spatially displacement comprises at least a rotation or a translation or scaling operation.

3. The encoding method according to claim 1, wherein said graph representation is generated basing on said sequence (f.sub.d) of a plurality of said sub-views organized in group of pictures (GOP) structures.

4. The encoding method according to claim 1, wherein in said sub-views graph generating phase the sub-views sequence (f.sub.d) is divided into multiple GOP consists of a predefined number G of sub-views.

5. The encoding method according to claim 3, wherein said representation of the sub-views sequence (f.sub.d) is generated by considering intra-view or inter-view correlations among the sub-views into the group of pictures (GOP) structure such that each node of said graph representation is connected to a predefined number of nearest nodes in terms of Euclidean distance within the same sub-view and the reference sub-view into the GOP structure.

6. The encoding method according to claim 1, wherein said sub-view generation phase comprises: a subaperture generating phase, wherein a subaperture image comprising a plurality of sub-views is generated, by composing each sub-view such that the pixel having the same relative position with respect to each macro-pixel center is used; a subaperture rearranging phase, wherein a sequence (f.sub.d) of sub-views is generated by rearranging the sub-views composing the generated subaperture image, on the basis of at least one predefined order.

7. The encoding method according to claim 6, wherein in said subaperture rearranging phase said predefined order is a raster scan order, or a helicoidal order, or a zig-zag order or a chess-like order.

8. The encoding method according to claim 1, wherein in said graph coding phase said bitstream (f.sub.d{circumflex over ()}) is generated by encoding said graph representation of the sub-views sequence (f.sub.d) according to the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT).

9. A method for decoding a bitstream comprising at least one encoded raw lenselet image (f), comprising: a receiving phase, wherein a graph-coded bitstream (f.sub.d{circumflex over ()}) of a sub-views sequence (f.sub.d) is received, wherein dummy pixels having undefined value are inserted into said raw lenselet image (f) and wherein each sub-view comprises pixels of the same angular coordinates extracted from different macro-pixels of said raw lenselet image (f), each macro-pixel comprising pixels corresponding to a specific view angle for the same point of a scene; an output phase, wherein the reconstructed light field image (f.sub.d{tilde over ()}) is outputted or displayed; a graph decoding phase, wherein said graph-coded bitstream (f.sub.d{circumflex over ()}) is decoded according to a predefined graph signal processing (GSP) technique, outputting a reconstructed sub-views sequence (f.sub.d), wherein said sub-views comprise dummy pixels situated in pixel locations having undefined color value; a demosaicing filter phase, wherein a full-color demosaiced lenselet image that includes the colors red, blue and green is generated through a color interpolation by applying a demosaicing technique to said sub-views sequence (f.sub.d); a raw lenselet rearrangement phase, wherein a full-color subaperture image (f.sub.d) that includes the colors red, blue and green is obtained basing on said demosaiced full-color lenselet image; wherein said output phase comprises outputting said generated full-color subaperture image (f.sub.d).

10. The decoding method according to claim 9, wherein said lenselet rearrangement phase comprises an image reconstructing phase wherein a reconstructed subaperture image comprising a plurality of sub-views is generated rearranging said reconstructed sub-view in the sequence (f.sub.d) on the basis of at least one predefined order; a lenselet image reconstructing phase, wherein a reconstructed lenselet image is generated such that the pixels of each sub-view are located in the corresponding macro-pixels on the basis of the order used in the encoding of the received raw lenselet image (f).

11. The decoding method according to claim 9, wherein in said lenselet rearrangement phase said predefined order is a raster scan order, or a helicoidal order, or a zig-zag order or a chess-like order.

12. The decoding method according claim 9, wherein in said graph decoding phase said reconstructed sub-views sequence (f.sub.d) is generated by decoding said graph representation of the sub-views sequence (f.sub.d) according to the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT).

13. An apparatus for encoding a raw lenselet image (f), comprising: an input unit configured for acquiring at least a portion of a raw lenselet image (f) from a source, comprising a plurality of macro-pixels, each macro-pixel comprising pixels corresponding to a specific view angle for the same point of a scene; an output unit configured for outputting at least a portion of a resulting bitstream (f.sub.d{circumflex over ()}); at least one processing unit configured for executing a set of instructions for encoding said raw lenselet image (f); a memory unit containing image data relating to said a raw lenselet image (f) and the result (f.sub.d, f.sub.d{circumflex over ()}) of the execution of said encoding instructions, wherein said at least one processing unit is configured for spatially displacing the pixels of said raw lenselet image (f) in a new multi-color image that includes the colors red, blue and green and has a larger number of columns and rows with respect to the received raw lenselet image, wherein dummy pixels are inserted in the pixel locations having undefined color channel value and wherein said displacement is performed so as to put the estimated center location of each macro-pixel onto integer pixel locations; wherein said at least one processing unit is further configured for generating a graph representation of a sequence of sub-views (f.sub.d) starting from said raw lenselet image (f), each sub-view comprising pixels of the same angular coordinates extracted from different macro-pixels of said raw lenselet image (f); wherein said at least one processing unit is further configured for fetching the graph representation of the sequence of sub-views (f.sub.d) from said memory unit, and for executing a graph signal processing (GSP) technique for coding said sequence of sub-views (f.sub.d), and storing the resulting bitstream (f.sub.d{circumflex over ()}) into said memory unit.

14. The encoding apparatus according to claim 13, wherein said at least one processing unit is further configured for: generating said sequence of sub-views (f.sub.d) by forming a subaperture image comprising a plurality of sub-views by composing each sub-view such that the pixel having the same relative position with respect to each macro-pixel center is used, and rearranging said sequence (f.sub.d) of sub-views composing the generated subaperture image on the basis of at least one predefined order.

15. The encoding apparatus according to claim 14, wherein said predefined order is a raster scan order, or a helicoidal order, or a zig-zag order, or a chess-like order.

16. The encoding apparatus according to claim 13, wherein said graph signal processing (GSP) technique is the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT).

17. An apparatus for decoding an encoded raw lenselet image comprising: an input unit configured to read a graph-coded bitstream (f.sub.d{circumflex over ()}) of a sub-views sequence (f.sub.d), wherein dummy pixels having undefined value are inserted into said raw lenselet image (f) and wherein each sub-view comprises pixels of the same angular coordinates extracted from different macro-pixels of said raw lenselet image (f), each macro-pixel comprising pixels corresponding to a specific view angle for the same point of a scene (f.sub.d{circumflex over ()}) from a communication channel or storage media, an output unit which reproduces or outputs the processed light field images or video streams (f.sub.d{tilde over ()}); at least one processing unit configured for executing a set of instruction for decoding said encoded images or video streams (f.sub.d{circumflex over ()}); a memory unit containing image data relating to said encoded images or video streams (f.sub.d{circumflex over ()}) and the result of the execution of said instructions for decoding; said at least one processing unit being configured for receiving and decoding the bitstream of the sub-views sequence (f.sub.d) according to a predefined graph signal processing (GSP) technique, such that a reconstructed sub-views sequence (f.sub.d) is recovered, wherein said sub-views comprise dummy pixels situated in pixel locations having undefined color value; said at least one processing unit being configured for receiving said reconstructed sub-views sequence (f.sub.d) and to generate a full-color demosaiced lenselet image that includes the colors red, blue and green through a color interpolation by applying a demosaicing technique to said sub-views.

18. The decoding apparatus according to claim 17, wherein said at least one processing unit is configured for: rearranging said sub-views on the basis of at least one predefined order for obtaining a reconstructed subaperture image (f.sub.d) comprising a plurality of sub-views, and; reconstructing a lenselet image such that the pixels of each sub-view are located in the corresponding macro-pixels on the basis of the order used in the encoding of the received raw lenselet image (f).

19. A non-transitory computer readable medium operable with a digital processing device, and which comprises portions of software code for executing the method according to claim 1.

Description

BRIEF DESCRIPTION OF DRAWING

(1) The characteristics and other advantages of the present invention will become apparent from the description of an embodiment illustrated in the appended drawings, provided purely by way of no limiting example, in which:

(2) FIG. 1 illustrates an exemplary light field cameras 100 and an exemplary plenoptic camera 150;

(3) FIG. 2 shows an example of lenselet image which is captured by plenoptic camera 150;

(4) FIG. 3 shows an example of subaperture image which is composed by a set of sub-views;

(5) FIG. 4 illustrates a block diagram of light field image encoding-decoding system according to the state of art;

(6) FIG. 5 illustrates the pre-processing operations for encoding light field images according to the state of art;

(7) FIG. 6 illustrates the pre-processing operations for encoding light field images according to an embodiment of the present invention;

(8) FIG. 7 shows the functioning of the pre-processing stage for encoding digital light field images according to an embodiment of the present invention;

(9) FIG. 8 shows the functioning of the demosaicing stage for decoding digital light field images according to an embodiment of the present invention;

(10) FIG. 9 illustrates a block diagram of light field image encoding-decoding system according to an embodiment of present invention;

(11) FIG. 10 shows a block diagram illustrating apparatuses for compressing digital light field image or video streams according to an embodiment of the present invention;

(12) FIG. 11 shows a block diagram illustrating apparatuses for decompressing digital light field image or video streams according to an embodiment of the present invention;

(13) FIG. 12 shows two examples of color filter array;

(14) FIG. 13 illustrates the sub-view generating process from macro-pixels array and vice versa;

(15) FIG. 14 shows an example of the green component of the generated subaperture image according to an embodiment of the present invention;

(16) FIG. 15 pictures an exemplary raster-scanner rearrangement of the sub-views which compose the generated subaperture image;

(17) FIG. 16 illustrates an example of the sub-views sequence subdivided in terms of GOPs structure;

(18) FIG. 17 shows the performance of a test encoder-decoder pair implemented according to an embodiment of the present invention; and

(19) FIG. 18 illustrates the operations for decoding light field images according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

(20) In this description, any reference to an embodiment will indicate that a particular configuration, structure or feature described in regard to the implementation of the invention is comprised in at least one embodiment. Therefore, the phrase in an embodiment and other similar phrases, which may be present in different parts of this description, will not necessarily be all related to the same embodiment. Furthermore, any particular configuration, structure or feature may be combined in one or more embodiments in any way deemed appropriate.

(21) The references below are therefore used only for sake of simplicity, and do not limit the protection scope or extension of the various embodiments.

(22) With reference to FIG. 10, an apparatus 1005 for compressing digital images or video streams (also named encoding apparatus 1005) comprises the following parts: a processing unit 1010, like a Central Processing Unit (CPU), configured for executing a set of instructions for carrying out a method for encoding digital images or video streams according to the invention (said method will be described in details in the following of present description); a memory unit 1030 of any kind (volatile or non volatile, removable or fixed) and technology (e.g. electronic, semiconductor-based or opto-electronic) containing data relating to the image to be compressed and preferably the instructions that implement the method for encoding digital images or video streams according to the invention, wherein the image data are a representation of at least a portion of said image and are preferably in binary format, and wherein said memory unit 1140 may also contain the result of the execution of instructions of said method; an input unit 1070 (e.g. I/O means), which can be configured by the CPU 1010 to read from a (video) source 1000 light field images or video streams to be processed; such input unit may for example include an adapter according to at least one of the following standards: USB, Firewire, RS232, IEEE 1284, Ethernet, Wi-Fi, or the like; a pre-processing unit 1020, which is configured for executing the phases of the method for generating a graph representation of a sequence of sub-views f.sub.d starting from a raw lenselet image f, which is generated by a light field image or video source 1000. In particular, this unit is configured for receiving the raw lenselet image from the source 1000, then transform it in order to put the estimated center location of each macro-pixel (composing the raw lenselet image) onto the integer pixel locations. Next, starting from the transformed raw lenselet image a subaperture image 670 is generated and then a sequence of sub-views (composing the subaperture image) is arranged on the basis of at least one predefined order, like a raster scan order, a helicoidal order, a zig-zag order, a chess-like order, and so on. Important is that the same order is used also in the decoding apparatus 1100. Finally, the sequence of sub-views is divided in terms of group of pictures GOPs from which a graph representation of the sequence of sub-views f.sub.d is obtained. a graph coding unit 1040, which is configured for fetching the graph representation of the sequence of sub-views f.sub.d from the memory unit 1030, executing the phases of the method for encoding digital images or video streams according to a graph signal processing (GSP) technique such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT) for coding each GOP separately, and storing the resulting bitstream f.sub.d{circumflex over ()} back into the memory unit 1030; output unit 1080 (e.g. a network or storage adapter), which can be configured by the CPU 1010 to transmit, through a communication channel, the processing result to a destination 1095 (e.g. a storage media, a remote client or the like); such output unit may for example include an adapter according to at least one of the following standards: Ethernet, SATA, SCSI, or the like; a communication bus 1090, which allows the exchange of information between the CPU 1010, the pre-processing unit 1020, the memory unit 1030, the graph coding unit 1040, the input unit 1070, and the output unit 1080. As an alternative to using the communication bus 1090, the CPU 1010, the pre-processing unit 1020, the memory unit 1030, the graph coding unit 1040, the input unit 1070, and the output unit 1080 can be connected by means of a star architecture.

(23) The video source 1000 can be either a provider of live images, such as a light field camera, or a provider of stored contents such as a disk or other storage and memorization devices. The Central Processing Unit (CPU) 1010 takes care of activating the proper sequence of operations performed by the units 1020, 1040, in the encoding process performed by the apparatus 1005.

(24) These units can be implemented by means of dedicated hardware components (e.g. CPLD, FPGA, or the like) or can be implemented through one or more sets of instructions which are executed by the CPU 1010; in the latter case, the units 1020, 1040 are just logical (virtual) units.

(25) When the apparatus 1005 is in an operating condition, the CPU 1010 first fetches the light field image f from the video source 1000 and loads it into the memory unit 1030.

(26) Next, the CPU 1010 activates the pre-processing unit 1020, which fetches the raw lenselet image f from the memory 1030, executes the phases of the method for pre-process the raw lenselet image f according to an embodiment of the invention (see FIG. 7), and stores the resulting graph representation of the sequence of sub-views f.sub.d back into the memory unit 1030.

(27) Successively, the CPU 1010 activates the graph coding unit 1040, which fetches from the memory 1030 the graph representation of the sequence of sub-views f.sub.d, executes the phases of the method for encode the sequence of sub-views f.sub.d according to a graph signal processing (GSP) techniques such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT), and stores the resulting bitstream f.sub.d{circumflex over ()} back into the memory unit 1030.

(28) At this point, the CPU 1010 may dispose of the data from the memory unit 1030 which are not required anymore at the encoder 1005.

(29) Finally, the CPU 1010 fetches the bitstream f.sub.d{circumflex over ()} from memory 1030 and puts it into the channel or saves it into the storage media 1195.

(30) With reference also to FIG. 11, an apparatus 1100 for decompressing digital images or video streams (also named decoding apparatus 1100) comprises the following parts: a processing unit 1110, like a Central Processing Unit (CPU), configured for executing a set of instruction for carrying out a method for decoding digital images or video streams according to the invention (said method will be described in details in the following of present description); a memory unit 1140 of any kind (volatile or non volatile, removable or fixed) and technology (e.g. electronic, semiconductor-based or opto-electronic) containing data relating to the compressed image received and preferably the instructions that implement the method for decompressing digital images or video streams according to the invention, wherein the image data are a representation of at least a portion of said image and are preferably in binary format, and wherein said memory unit 1240 may also contain the result of the execution of instructions of said method; an input unit 1180 (e.g. a network or storage adapter), which can be can be configured by the CPU 1110 to read the encoded images or video streams from a communication channel or storage media 1095; said input unit 1180 may for example comprise an adapter according to at least one of the following standards: Ethernet, SATA, SCSI, or the like; a graph decoding unit 1120, which is configured for executing the phases of the method for decompressing digital light field images or video streams according to the invention; in particular, this unit is configured for receiving and decoding the bitstream f.sub.d{circumflex over ()} of the sub-views sequence according to a predefined graph signal processing (GSP) technique, such that the reconstructed sub-views sequence f.sub.d is recovered; a demosaicing unit 1150 which is configured for executing the phases of the method for receiving the reconstructed sub-views sequence f.sub.d and generate a full-color subaperture image f.sub.d according to an embodiment of the invention (see FIG. 8 and FIG. 18). In particular, starting from the reconstructed sub-views sequence f.sub.d a reconstructed subaperture image is obtained by re-arranging the reconstructed sub-views sequences according to at least one predefined order, then a reconstructed lenselet image is generated by locating the pixels of each sub-view into the corresponding macro-pixels 650 on the basis of their order. Successively, a demosaicing technique is applied to the reconstructed lenselet image in order to generate the full-color lenselet image through a RGB color interpolation. Finally, starting from the transformed full-color lenselet image, a full-color subaperture image f.sub.d is generated such that each sub-view comprising said full-color subaperture image f.sub.d is composed by considering the same relative pixel position with respect to each macro-pixel center; a post-processing unit 1130 which is configured for receiving the full-color subaperture image f.sub.d and for generating a reconstructed light field image f.sub.d{tilde over ()}. In particular, by using techniques which enables operation such as image re-focusing, noise reduction, 3D view construction and modification of depth of field; an output video unit 1170, like a video adapter, which can be configured by the CPU 1110 to reproduce and/or output, preferably on a display 1195, the processed (decoded or decompressed) light field images or video streams; said output video unit may for example include an adapter according to at least one of the following standards: VGA, S-video, HDMI, Ethernet or the like; a communication bus 1190, which allows the exchange of information between the CPU 1110, the graph decoding unit 1120, the demosaicing unit 1150, the memory unit 1140, the output video unit 1170, and the input unit 1180. As an alternative to using the communication bus 1190, the CPU 1110, the graph decoding unit 1120, the demosaicing unit 1150, the memory unit 1140, the output video unit 1170, and the input unit 1180 can be connected by means of a star architecture.

(31) As for the previously described encoding apparatus 1005, also the CPU 1110 of the decoding apparatus 1100 takes care of activating the proper sequence of operations performed by the units 1120, 1130 and 1150.

(32) These units can be implemented by means of dedicated hardware components (e.g. CPLD, FPGA, or the like) or can be implemented through one or more sets of instructions stored in a memory unit which are executed by the CPU 1110; in the latter case, the units 1120, 1130 and 1150 are just a logical (virtual) units.

(33) When the apparatus 1100 is in an operating condition, the CPU 1110 first fetches the bitstream f.sub.d{circumflex over ()} from the channel or storage media 1095 via any possible input unit and loads it into the memory unit 1140.

(34) Then, the CPU 1110 activates the graph decoding unit 1120, which fetches from the memory 1140 the bitstream f.sub.d{circumflex over ()}, executes phases of the method for decoding the bitstream f.sub.d{circumflex over ()} of the sub-views sequence according to a predefined graph signal processing (GSP) technique, such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT), outputs the reconstructed sub-views sequence f.sub.d, and loads it into the memory unit 1140.

(35) Any GSP technique can be used according to the invention; important is that the same technique is used in the encoding and decoding apparatus 1100 for assuring a correct reconstruction of the original light field image.

(36) Successively, the CPU 1110 activates the demosaicing unit 1150, which fetches from the memory 1140 the reconstructed sub-views sequence f.sub.d, and executes phases of the method for generating a full-color subaperture image f.sub.d according to the invention, and loads it into the memory unit 1140.

(37) Then, the CPU 1110 activates the post-processing unit 1130, which fetches from the memory 1140 the full-color subaperture image f.sub.d and generates a reconstructed light field image f.sub.d{tilde over ()}, storing it into the memory unit 1140.

(38) At this point, the CPU 1110 may dispose of the data from the memory which are not required anymore at the decoder side.

(39) Finally, the CPU 1110 fetches from memory 1140 the recovered light field image f.sub.d{tilde over ()} and sends it, by means of the video adapter 1170, to the display unit 1195.

(40) It should be noted how the encoding and decoding apparatuses described in the figures may be controlled by the CPU 1110 to internally operate in a pipelined fashion, enabling to reduce the overall time required to process each image, i.e. by performing more instructions at the same time (e.g. using more than one CPU and/or CPU core).

(41) It should also be noted than many other operations may be performed on the output data of the coding device 1005 before sending them on the channel or memorizing them on a storage unit, like modulation, channel coding (i.e. error protection).

(42) Conversely, the same inverse operations may be performed on the input data of the decoding device 1100 before effectively process them, e.g. demodulation and error correction. Those operations are irrelevant for embodying the present invention and will be therefore omitted.

(43) Besides, the block diagrams shown in FIG. 10 and FIG. 11 are of exemplificative nature only; they allow to understand how the inventions works and how it can be realized by the person skilled in the art.

(44) The skilled person understands that these charts have no limitative meaning in the sense that functions, interrelations and signals shown therein can be arranged in many equivalents ways; for example, operations appearing to be performed by different logical blocks can be performed by any combination of hardware and software resources, being also the same resources for realizing different or all blocks.

(45) The encoding process and the decoding process will now be described in detail.

(46) Encoding

(47) In order to show how the encoding process occurs, it is assumed that the image f (or a block thereof) to be processed is preferably a color patterned raw lenselet image, where each pixel is encoded over 8 bit so that the value of said pixel can be represented by means of an integer value ranging between 0 and 255. Of course, this is only an example; images of higher color depth (e.g. 16, 24, 30, 36 or 48 bit) can be processed by the invention without any loss of generality.

(48) The image f can be obtained applying a color filter array on a square grid of photosensors (e.g. CDD sensors); a well-known color filter array is for example the Bayer filter, which is used in most single-chip digital image sensors.

(49) FIG. 12 shows some examples of color filter array, where the letters R,G,B indicate respectively the red, green and blue color filters which are applied on a grid of photosensors.

(50) With also reference to FIG. 9, it is now described how the different parts of the encoding apparatus 900 interact for compressing digital light field images or video streams.

(51) With also reference to FIG. 6 and FIG. 7 the pre-processing unit 920 preferably comprises the following steps: a raw-lenselet image receiving step 705 for receiving a color patterned raw lenselet image f, see FIG. 6(a), which is generated by photosensor 410; no demosaicing of the raw image is performed before the subsequent operation; a transform step 710 for translating, rotating and scaling at least a portion of said raw lenselet image f obtained in the previous step; in particular, the pixels of the raw lenselet image f are spatially displaced (that can comprise scaling, and/or translations and/or rotations) in a new multi-color image having a larger number of columns and rows with respect to the received raw lenselet image, wherein dummy pixels e.g. 610, 620 are inserted in the pixel locations having undefined color channel value and wherein said displacement is performed so as to put the estimated center location of each macro-pixel 650 (denoted in dashed line, FIG. 6(b)) onto integer pixel locations. For example, this operation can be performed as described by D. G. Dansereau, O. Pizarro and S. B. Williams in Decoding, calibration and rectification for lenselet-based plenoptic cameras, published in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013); preferably: a subaperture generating step 715 for generating a subaperture image 670 from the transformed raw lenselet image, such that each sub-view comprising said subaperture image is composed by considering the same relative pixel position with respect to each macro-pixel center. FIG. 13 illustrates the sub-view generating process: the pixels 1360 and 1370, which compose a sub-view 1310, are taken from the macro-pixels 1320 and 1340 respectively, such that they have the same relative position with respect to their corresponding macro-pixels center 1330 and 1350 respectively. FIG. 14(a) shows an example of the green component of the generated subaperture image 670, while FIG. 14(b) is a detail of FIG. 14(a). It should be noted that due the presence of dummy pixels (e.g. 610, 620), the pixels which represent the image information (e.g. 630, 640) are irregularly spaced on the generated subaperture image 670, see FIG. 6(c) and FIG. 14(b), thus cannot be encoded by the traditional transform coding such as discrete cosine transform (DCT) or discrete wavelet transform (DWT), which are approached for example in HEVC standard. The irregular spacing is due to the missing demosaicing operation performed in the state of the art, which renders ineffective the traditional coding approaches. However, in order to encode the generated subaperture image, its graph representation and the related graph-based compression techniques can be used instead; in fact, the irregular spacing of the pixels generates discontinuities in the image, i.e. high frequency components which makes the traditional coding techniques inefficient with respect to the GSP techniques; in addition, an image with less pixel results in a simpler graph representation, that can be compressed more efficiently; preferably: a subaperture rearranging step 720 for rearranging in a sequence the sub-views composing the generated subaperture image 670, on the basis of at least one predefined order. FIG. 15 shows an exemplary raster-scanner rearrangement of the sub-views e.g. 1510 composing the generated subaperture image 1530, obtaining a sub-view sequence f.sub.d in horizontal scan order; a sub-views graph generating step 725 for generating a graph representation of the sub-view images sequence in order to perform a graph based compression of said sub-view images sequence, i.e. any GSP technique.

(52) Two distinctive schemes for graph connection can be considered.

(53) The first scheme takes into account only intra-view connections when constructing a graph, where each node is connected to a predefined number K of nearest nodes in terms of Euclidean distance, i.e. the distance between available irregularly spaced pixels (e.g. 630, 640) within the same sub-view of the sequence.

(54) The second scheme takes into account both intra and inter-view correlations among the sub-views of the sequence.

(55) In order to reduce graph complexity, the sub-views sequence is divided into multiple GOPs consists of a predefined number G of sub-views.

(56) FIG. 16 shows an example of the sub-views sequence f.sub.d subdivided in terms of GOPs structure 1610, which is composed by four sub-views comprising a reference sub-view 1620.

(57) Successively, a sub-view matching for motion estimation between each sub-view and the previous reference sub-view is performed in the sequence.

(58) The optimal global motion vector can be determined for each sub-view in terms of sum of squared error (SSE), which can be evaluated considering the pixel samples of each sub-view and the previous reference sub-view.

(59) The matching is considered for the whole sub-view, instead of applying the block-based matching employed for example for the motion estimation in HEVC.

(60) Specifically, each mn sub-view is first extrapolated to the size of (m+2r)(n+2r) before motion search, where r is the motion search width.

(61) This reduces the overhead in encoding of the motion vectors. The sub-view extrapolation can be performed by employing several techniques, for example by copying the border pixel samples of each sub-view.

(62) After motion estimation, each pixel is connected to a predefined number P of nearest neighbours in terms of Euclidean distance within the same sub-view and the reference view shifted by the optimal motion vector.

(63) With also reference to FIG. 9 the graph coding unit 940 (and the decoding apparatus 1100 as well, for decoding) encodes the sub-views sequence using graph signal processing (GSP) techniques such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT) for coding each GOP separately.

(64) A graph G=(E,V) is composed of a set of nodes vV, connected with links. For each link e.sub.i,jE, connecting nodes v.sub.i and v.sub.j, there is an associated weight of non-negative value w.sub.ij [0,1], which captures the similarity between the connected nodes.

(65) An image f can be represented as a graph where the pixels of the image correspond to the graph nodes, while the weights of the links describe the pixels similarity which can be evaluated using a predetermined non-linear function (e.g. Gaussian or Cauchy function) depending on the grayscale space distance d.sub.i,j=|f.sub.if.sub.j| between the i-th pixel f.sub.i and the j-th pixel f.sub.j of the image.

(66) In the Graph Fourier transform (GFT) technique, the graph information can be represented with a weights matrix W which elements are the weights w.sub.ij of the graph, then the corresponding Laplacian matrix can be obtained as L=D-W where D is a diagonal matrix with elements d.sub.i=.sub.kw.sub.ik. The GFT is performed by the mathematical expression {circumflex over (f)}=U.sup.Tf where U is the matrix which columns are the eigenvectors of the matrix L, and f is the raster-scanner vector representation of the image f.

(67) The coefficients {circumflex over (f)} and the weights w.sub.ij are then quantized and entropy coded. More related work known in the art describe approaches improving the GFT based coding, as shown for example by W. Hu, G. Cheung, A. Ortega, and O. C. Au in Multiresolution graph Fourier transform for compression of piecewise smooth images, published in IEEE Transactions on Image Processing.

(68) The Graph based Lifting Transform (GLT) technique is a multi-level filterbank that guarantees invertibility. At each level m, the graph nodes are first divided into two disjoint sets, a prediction set SP.sup.m and an update set SU.sup.m.

(69) The values in SU.sup.m are used to predict the values in SP.sup.m, the resulting prediction errors are stored in SP.sup.m, and are then used to update the values in SU.sup.m.

(70) The smoothed signal in SU.sup.m will serve as the input signal to level m+1, while the computation for coefficients in SP.sup.m uses only the information in SU.sup.m, and vice versa.

(71) Carrying on the process iteratively produces a multi-resolution decomposition. For video/image compression applications, the coefficients in the update set SU.sup.M of the highest-level M will be quantized and entropy coded. More related work known in the art describe approaches improving the GLT based coding, as shown for example by Y.-H. Chao, A. Ortega, and S. Yea, Graph-based lifting transform for intra-predicted video coding, published in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2016).

(72) Summarizing, with also reference to FIG. 9 and FIG. 10, the method for encoding digital images according to the invention comprises the following phases: a receiving phase, wherein at least a portion of a color patterned raw lenselet image f is received by means of the input unit 1070; a transform phase, wherein the color patterned raw lenselet image f is rotated, translated and scaled by the unit 920 in order to put, through the processing unit 1010, the estimated center location of each macro-pixel onto the integer pixel locations; preferably: a subaperture generating phase, wherein a subaperture image 670 is generated from the transformed raw lenselet image, by the unit 920 by composing each sub-view, through the processing unit 1010, such that the pixel having the same relative position with respect to each macro-pixel center is used; preferably: a subaperture rearranging phase, wherein a sequence of sub-views is generated by the unit 920 by rearranging through the processing unit 1010 the sub-views composing the generated sub aperture image, on the basis of at least one predefined order (for example the raster scan order as depicted in FIG. 15); preferably: a sub-views graph generating phase, wherein a graph representation of the sub-view images sequence is generated by the unit 920 by organizing said sub-views sequence in terms of GOP structures and by considering intra-view or inter-view correlations among the sub-views into the GOP structure through the processing unit 1010, such that each node of said graph representation is connected to a predefined number of nearest nodes in terms of Euclidean distance within the same sub-view and the reference sub-view into the GOP structure; a graph coding phase, wherein a bitstream f.sub.d{circumflex over ()} is generated by the unit 940 by encoding through the processing unit 1110, the graph representation of the sub-view images sequence according to a predefined graph signal processing (GSP) technique (step 730).

(73) Finally, the graph-coded bitstream f.sub.d{circumflex over ()} of the sub-views sequence can be transmitted and/or stored by means of the output unit 1080.

(74) Decoding

(75) With reference to FIG. 8 and FIG. 9, the decoder 960 comprises the graph decoding unit 970, the demosaicing unit 975 and the post-processing unit 980.

(76) The graph decoding unit 970 is configured to receive and decode the bitstream f.sub.d{circumflex over ()} of the sub-views sequence according to a predefined graph signal processing (GSP) techniques, outputting the reconstructed sub-views sequence f.sub.d (step 805).

(77) The demosaicing unit 975 preferably performs the following steps: a subaperture image reconstructing step 810 for receiving the reconstructed sub-views sequence f.sub.d and to generate the reconstructed subaperture image f.sub.d, rearranging each reconstructed sub-view in the sequence on the basis of at least one predefined order (see FIG. 15); a lenselet image reconstructing step 815 for receiving the reconstructed subaperture image f.sub.d and to generate the reconstructed lenselet image, in particular the pixels of each sub-view 1310 are located in the corresponding macro-pixels 1320, 1340 on the basis of their order, as pictured in FIG. 13; a demosaicing filter step 820 for receiving the reconstructed lenselet image and to apply a demosaicing algorithm in order to generate the full-color lenselet image through a RGB color interpolation; preferably: a subaperture image generating step 830 for receiving the transformed full-color lenselet image and to generate a full-color subaperture image f.sub.d, such that each sub-view 1310 comprising said full-color subaperture image f.sub.d is composed by considering the same relative pixel position with respect to each macro-pixel center 1330, 1350 (see FIG. 13); this operation is performed on each color channel of the transformed full-color lenselet image.

(78) The optional post-processing unit 980 is configured to receive the full-color subaperture image f.sub.d and to generate a reconstructed light field image f.sub.d{tilde over ()}, using operations permitted in the light field images such as re-focusing, noise reduction, 3D view construction and modification of depth of field.

(79) Summarizing, with also reference to FIG. 9 and FIG. 11, the method for decoding digital images or video streams according to the invention comprises the following phases: a receiving phase, wherein the bitstream f.sub.d{circumflex over ()} of the sub-views sequence is received by means of the input unit 1180; a graph decoding phase, wherein the bitstream f.sub.d{circumflex over ()} is decoded, by means of the processing unit 1110, according to a predefined graph signal processing (GSP) technique, outputting the reconstructed sub-views sequence f.sub.d; preferably: a subaperture image reconstructing phase, wherein a reconstructed subaperture image f.sub.d is generated by means of the processing unit 1110, rearranging each reconstructed sub-view in the sequence, on the basis of at least one predefined order; preferably: a lenselet image reconstructing phase, wherein a reconstructed lenselet image is generated by means of the processing unit 1110 such that, the pixels of each sub-view are located in the corresponding macro-pixels on the basis of the order used in the coding method; a demosaicing filter phase, wherein a full-color lenselet image is generated by means of the processing unit 1110, through a RGB color interpolation by applying a demosaicing technique; preferably: a subaperture image generating phase, wherein a full-color subaperture image f.sub.d is generated by means of the processing unit 1110, such that each sub-view comprising said full-color subaperture image f.sub.d is composed by considering the same relative pixel position respect each macro-pixel center for each color channel of said full-color lenselet image; preferably: a post-processing phase, wherein a reconstructed light field image f.sub.d{tilde over ()} is generated by means of the processing unit 1110, by using operations allowed in the light field images such as re-focusing, noise reduction, 3D view construction and modification of depth of field starting from the full-color subaperture image f.sub.d.

(80) Finally, the reconstructed light field image f.sub.d{tilde over ()} can be outputted by means of output video unit 1170 and displayed on the display unit 1195.

(81) With reference to FIG. 15, FIG. 16 and FIG. 17, the results of performance tests conducted by the inventors are going to be discussed. In this test, an encoder-decoder pair implemented according to an embodiment of the present invention has been evaluated.

(82) In order to perform the coding-encoding test, the EPFL database (M. Rerabek and T. Ebrahimi, New Light Field Image Dataset, in 8th International Conference on Quality of Multimedia Experience (QoMEX), no. EPFL-CONF-218363, 2016) was used.

(83) The subaperture image consists of 193 sub-views of size 432624. FIG. 17 shows the performance of the method described in an embodiment of the present invention, which is compared with the coding approach using HEVC standard.

(84) The ordinate axis denotes the average PSNR for R, G, and B color components. Compared to state-of-the-art schemes, a coding gain is achieved at the high-bitrate region.

(85) For the test, both All-intra and Low delay P configurations were used for the baseline HEVC based scheme.

(86) For Low delay P configuration in HEVC. The sub-views are arranged into pseudo-sequence in the same way as pictured in FIG. 15, and divided into multiple size 4 GOPs (FIG. 16).

(87) The first view in each GOP is compressed as an I-frame, and the remaining frames are coded as P-frames. For the proposed graph based approach, each node is connected to 6 nearest neighbours, and the search width r=2 for sub-view matching.

(88) The transformed coefficients are uniformly quantized and entropy coded using the Alphabet and Group Partitioning (AGP) proposed by Said and Pearlman in A new, fast, and efficient image codec based on set partitioning in hierarchical trees, published in IEEE Transactions on circuits and systems for video technology, vol. 6, no. 3, pp. 243-250, 1996. In order to evaluate the reconstructed lenselet image, using graph based coding, the reconstructed lenselet image is demosaiced and converted to the colored subaperture image in a same way as proposed by D. G. Dansereau, O. Pizarro, and S. B. Williams, in Decoding, calibration and rectification for lenselet-based plenoptic cameras, published in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013.

(89) In the baseline method, the reconstructed YUV 4:2:0 sequences are converted to RGB 4:4:4, where the upsampling for U and V components is based on nearest neighbour. Concluding, the obtained results show that the method described in the present invention can outperform the state-of-the-art schemes like a HEVC-based approach.

(90) In an alternative embodiment of the invention, the patterned raw lenselet image f can be generated by employing other color filter arrays placed on a square grid of photosensors, besides the well-known Bayer filter.

(91) In another embodiment of the invention, the patterned raw lenselet image f can be generated by capturing other combinations of color components, for example RGBY (red, green, blue, yellow) instead of RGB.

(92) In other embodiments, the invention is integrated in a video coding technique wherein also the temporal correlation between different light field images is taken into account. To that end, a prediction mechanism similar to those used in the conventional video compression standards can be used in combination with the invention for effectively compressing and decompressing a video signal.

(93) In other embodiments, the encoding and decoding stages described in the present invention can be performed employing other graph signal processing (GSP) techniques instead of the Graph Fourier transform (GFT), or the Graph based Lifting Transform (GLT).

(94) In other embodiments, the graph signal processing (GSP) technique employed at the encoding and decoding stages can be signalled from the encoder apparatus to the decoder apparatus. Alternatively, the GSP technique employed by both the encoder and decoder is defined in a technical standard.

(95) The present description has tackled some of the possible variants, but it will be apparent to the man skilled in the art that other embodiments may also be implemented, wherein some elements may be replaced with other technically equivalent elements. The present invention is not therefore limited to the explanatory examples described herein, but may be subject to many modifications, improvements or replacements of equivalent parts and elements without departing from the basic inventive idea, as set out in the following claims.