Compression of a raw image

Abstract

The invention relates to a raw image encoder for encoding a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, the raw image encoder configured to subject the raw image to a color transformation to obtain a color image, by mapping, for each pixel cell block, the four pixels of the pixel cell block onto a color component quadruple forming a sample of the color image, the color component quadruple comprising three color component values of a target color space, and one pseudo color component value, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.

Claims

1. Raw image encoder for encoding a Bayer pattern raw image, configured to map each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), G.sub.1 (green), G.sub.2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=\left[\begin{array}{cccc}3/4& -1/4& -1/4& -1/4\\ -1/4& -1/4& -1/4& 3/4\\ 1/4& 1/4& 1/4& 1/4\\ 0& 1& -1& 0\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]\mathrm{or}\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=a.Math.\left[\begin{array}{cccc}3/4& -1/4& -1/4& -1/4\\ -1/4& -1/4& -1/4& 3/4\\ 1/4& 1/4& 1/4& 1/4\\ 0& 1& -1& 0\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]$ with a being a constant, which forms a sample of the color image, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.

2. Raw image encoder according to claim 1, wherein the multi-component picture encoding comprises lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.

3. Decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{or}\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=b.Math.\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]$ with b being a constant, onto pixels R (red), G.sub.1 (green), G.sub.2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image.

4. Decoder according to claim 3, the multi-component picture decoding comprises spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.

5. Decoder according to claim 3, configured to, in a coding mode option, reconstruct from the compressed data stream an RGB image, by decoding from the compressed data stream by multi-component picture decoding the color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 0\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{or}\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=b.Math.\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 0\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]$ with b being a constant onto R (red), G (green) and B (blue) components of a sample an RGB image.

6. Decoder for reconstructing from a compressed data stream an RGB image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 0\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{or}\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=b.Math.\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 0\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]$ with b being a constant onto R (red), G (green) and B (blue) components of a sample an RGB image.

7. Method for encoding a Bayer pattern raw image, comprising mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), G.sub.1 (green), G.sub.2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=\left[\begin{array}{cccc}3/4& -1/4& -1/4& -1/4\\ -1/4& -1/4& -1/4& 3/4\\ 1/4& 1/4& 1/4& 1/4\\ 0& 1& -1& 0\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]\mathrm{or}\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=a.Math.\left[\begin{array}{cccc}3/4& -1/4& -1/4& -1/4\\ -1/4& -1/4& -1/4& 3/4\\ 1/4& 1/4& 1/4& 1/4\\ 0& 1& -1& 0\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]$ with a being a constant which forms a sample of the color image, and subjecting the color image to a multi-component picture encoding to obtain a compressed data stream.

8. Method for reconstructing from a compressed data stream a Bayer pattern raw image, comprising decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{or}\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=b.Math.\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]$ with b being a constant, onto pixels R (red), G.sub.1 (green), G.sub.2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image.

9. A non-transitory digital storage medium having a computer program stored thereon to perform the method for reconstructing from a compressed data stream a Bayer pattern raw image, comprising decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to $\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{or}\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=b.Math.\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]$ with b being a constant, onto pixels R (red), G.sub.1 (green), G.sub.2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, when said computer program is run by a computer.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:

(2) FIG. 1 shows schematically a pixel cell block of an image sensor subjected to a demosaicing step;

(3) FIG. 2 illustrates schematically a first and a second apparatus configured to perform color transforming a color image, according to a first and a second embodiment of the present invention;

(4) FIG. 3 illustrates schematically a raw image encoder, according to a specific embodiment of the present invention; and

(5) FIG. 4 illustrates schematically a decoder, according to a specific embodiment of the present invention;

DETAILED DESCRIPTION OF THE INVENTION

(6) The subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It is understood that subject matter embodiments can be practiced without these specific details.

(7) In particular, the description starts with presenting a specific example, sets out advantages and then provides further embodiments rendering clear that the advantages may also be achieved by slightly deviating from the specific example outlined before.

(8) Many cameras today are built with only one image sensor chip for image capturing. These sensors are equipped with color filters on top of the photosensitive pixel cells to capture multiple light spectra or color components. FIG. 1 shows schematically a pixel cell block of a commonly used image sensor equipped with a color filter pattern, the so called Bayer pattern. This pattern is a set of (one) red, (two) green and (one) blue color filters (RGB filters) on top of a pixel cell block formed by four pixels, each to create one red, one blue and two green sensitive pixels. The captured values from these pixel cells are the so called raw data. However for displaying a color image at least 3 color values (RGB) for each pixel cell position are required. For this reason the raw data may be processedthis process is called demosaicingto three color components values (typically RGB or YC.sub.bC.sub.r values) for each pixel cell position. This can be done by linear interpolation.

(9) An effective compression of Bayer raw data is, however, feasible as outlined now.

(10) The first step in compressing an RGB color image is typically a color transformation, as the information in the color components are strongly correlated and this redundancy should be minimized. The result is typically a luminance component and two color difference components YUV or YC.sub.bC.sub.r.

(11) To compress Bayer raw data also a color decorrelation should be used. In comparison to standard RGB images however we can use the neighbored four color components R, G.sub.1, G.sub.2 and B on slightly different positions. The color decorrelation itself should be reversible to recreate the original raw image. If we neglect the slightly different position, we can use the following formula:

(12) $\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=\left[\begin{array}{cccc}{m}_{11}& m_{12}& m_{13}& m_{14}\\ m_{21}& m_{22}& m_{23}& m_{24}\\ m_{31}& m_{32}& m_{33}& m_{34}\\ m_{41}& m_{42}& m_{43}& m_{44}\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]$

(13) The decorrelated components are now one Red-Chrominance Cr, one Blue-Chrominance Cb, one Luminance Y and one pseudo Delta-Luminance DY.

(14) The following decorrelation has multiple advantages.

(15) $\mathrm{Cr}=R-Y$$R=C_r+Y$$C_b=B-Y$$B=C_b+Y$$Y=\left(R+G_1+G_2+B\right)/4$$G_1=-1/2C_r-1/2C_b+Y+1/2\mathrm{DY}$$\mathrm{DY}=G_1-G_2$$G_2=G_1-\mathrm{DY}$$\begin{array}{cc}\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]=\left[\begin{array}{cccc}3/4& -1/4& -1/4& -1/4\\ -1/4& -1/4& -1/4& 3/4\\ 1/4& 1/4& 1/4& 1/4\\ 0& 1& -1& 0\end{array}\right].Math.\left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]& \left(1\right)\\ \left[\begin{array}{c}R\\ G_1\\ G_2\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 1/2\\ -1/2& -1/2& 1& -1/2\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]& \left(2\right)\end{array}$

(16) Firstly, it is reversible. That means the four components C.sub.r, C.sub.b, Y and DY can be transformed back lossless to R, G.sub.1, G.sub.2, B. Only a lossy compression between forth and back transformation would result in losses.

(17) Secondly, the first three components C.sub.r, C.sub.b, Y can be used directly to transform them into an RGB image with half width and half height by using G as mean value of G.sub.1 and G.sub.2. This is equivalent to consider one Bayer pixel cell block as one RGB macro pixel.

(18) 0$\begin{array}{cc}\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[\begin{array}{cccc}1& 0& 1& 0\\ -1/2& -1/2& 1& 0\\ 0& 1& 1& 0\end{array}\right].Math.\left[\begin{array}{c}{C}_r\\ C_b\\ Y\\ \mathrm{DY}\end{array}\right]\mathrm{with}G=\frac{G_1+G_2}{2}& \left(3\right)\end{array}$

(19) Third, the three components C.sub.r, C.sub.b and Y can be considered as color image with a residual DY. By using all 4 components the raw image can be reconstructed.

(20) Summarizing, using the above color transform for compression of raw data we are able to achieve high compression efficiency owing to the good decorrelation behavior. Nevertheless, it is easy to use the transformed C.sub.r, C.sub.b, Y data after transformation as RGB image with half size and half width without greater efforts. The residual DY and the half resolution image together may be used to reconstruct the Bayer raw data to full resolution image.

(21) FIG. 2 illustrates schematically a color transformation and its retransformation which underlies embodiments for a raw image encoder and its pendant, the decoder, embodiments of which are further outlined below.

(22) The forward color transform 5 is for color transforming of a raw image 10 into a color image 18. The color image 10 is formed by a number of X lines by Y columns of pixel cell blocks 12, wherein each cell blocks 12 comprises four pixels 14, arranged as a 22 sub-block. That is, the raw image 10 has N times M pixels clustered into N/2 times M/2 superpixels, called pixel cell blocks 12. To compress the raw data in image 10 which might correspond to a Bayer pattern, a color decorrelation is realized by the color transformation 5. In comparison to standard multi-component images such as image 18 and such as RGB images, for example, the neighbored four color components R, G.sub.1, G.sub.2 and B in a pixel cell block 12 of raw image 10 reside on slightly different positions.

(23) The color transform 5 is designed to deal with a raw image 10 of such a color pattern. As said, the raw image 10 is partitioned into pixel cell blocks 12, each pixel cell block 12 comprising four pixels 14, each pixel 14 being associated with one of three base colors so that each pixel cell block 12 comprises, for each of the three base colors, at least one pixel. For each pixel cell block 12, the four pixels 14 of the pixel cell block 12 are mapped onto a color component quadruple 16 forming a sample of the color image 18.

(24) The mapping 5 may be done according to (1): each pixel cell block 12 of a Bayer pattern raw image 10 which comprises four pixels R red, G.sub.1 green, G.sub.2 green, and B blue, is mapped onto a color component quadruple {C.sub.r, C.sub.b, Y, DY} 16 according to (1), with the quadruple forms a sample of the color image 18.

(25) The color component quadruple 16 comprises three color component values 20 of a target color spaces and one pseudo color component value 22. The decorrelated components color component quadruple 16 may be referred as e.g. color component values 20 being one Red-Chrominance C.sub.r, one Blue-Chrominance C.sub.b, one Luminance Y as well as the pseudo color component value 22 as a pseudo Delta-Luminance DY. That is, the afore-mentioned target color space may be elected to be a YC.sub.r C.sub.b color space. The first three component values (20) C.sub.r, C.sub.b, Y can be used e.g. to be directly transformed into an RGB image with half width and half height. This is equivalent to consider one pixel cell block 12 as one RGB macro pixel. However, alternatives using other color spaces are feasible as well

(26) The four color components Cr, Cb, Y and DY may be transformed back lossless to the color components R, G.sub.1, G.sub.2 and B. This is done by the color retransformation 25. Function (1) and (2) may, thus, be based on a linear mapping. In transform 5 each of the three color component values 20 may result from a weighted sum of the four pixels 14 of the pixel cell block 12. The weights of the weighted sum of the two pixels of the pixel cell block may be associated accordingly with the same base color are equal to each other. See, for instance, the middle columns in the first three lines of the matrix of (1). Further, the linear mapping 5 may be such that the one pseudo color component value 22 results from a further weighted sum of the two pixels of the pixel cell block associated with the same base color, wherein only the weights of the weighted sum of the two pixels being of opposite sign. See the lower row in (1), for instance.

(27) FIG. 2 illustrates schematically also the color retransforming procedure 25 from color image 18 to raw image 10. The transform 25 maps, for each sample of the color image 18, the color component quadruple 16 representing this sample, onto the four pixels 14 of the corresponding pixel cell block 12. That is, image 18 comprises as many samples as raw image 10 has pixel cell blocks 12. The color component quadruple 16 comprises three color component values 20 of a predetermined color space and a pseudo color component value 22.

(28) In the example set out below, the retransform 25 maps each color component quadruple {C.sub.r, C.sub.b, Y, DY} 16 according to (2) onto pixels R red, G.sub.1 green, G.sub.2 green, and B blue of a corresponding pixel cell block 12 of the Bayer pattern raw image 10. Again, other examples would be feasible as well. In particular, the mapping may be a linear mapping where the two pixels of the pixel cell block 12 being associated with the same base color respectively, i.e. G.sub.1 and G.sub.2, result from a weighted sum of the color component quadruple 16 with weights of the weighted sum for the two pixels being equal. See the first three columns in the middle two lines of the matrix in (2), for instance. The weights of the weighted sum of the one pseudo color component value may be of opposite sign. See the rightmost column in the middle two lines of the matrix in (2), for instance. The predetermined color space may be selected to be a YC.sub.r C.sub.b color space or differently.

(29) FIG. 3 illustrates schematically an encoding device 28 comprising the raw image encoder 5 and a multi-component picture encoder 30 configured to encode a color image 18 output by the apparatus into a compressed multi-component data stream 32. The multi-component picture encoding performed may comprise lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.

(30) FIG. 4 illustrates schematically a raw image decoding device 40 configured for reconstructing a raw image 10 from a multi-component data stream 32 comprising a decoder 25 and a multi-component picture decoder 42 configured to decode a color image 18 from the multi-component data stream 32. The multi-component picture decoding may comprise spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.