Face region detection based light field video compression

Abstract

A method of perceptual video coding based on face detection is provided. The method includes calculating a bit allocation scheme for coding a light field video based on a saliency map of the face, calculating an LCU level Lagrange multiplier for coding a light field video based on a saliency map of the face and calculating an LCU level quantization parameter for coding a light field video based on a saliency map of the face.

Claims

1. A method of light field video coding, comprising: determining a region of interest in a light field video; assigning a visual attention value to the region of interest, wherein the visual attention value is different from that for other regions of the light field video; calculating a bit allocation scheme based on the visual attention value of the region of interest, wherein the bit allocation scheme is calculated in accordance to: $\begin{array}{c}{R}_n^{\prime }=\frac{A_n}{\underset{i=1}{\overset{N}{.Math.}}{A}_i}*R_{\mathrm{frame}}\\ A_n=\frac{\underset{i=1}{\overset{N_b}{.Math.}}{f}_i}{N_b}\end{array}$ wherein R.sub.frame is the total bits of a video frame, N is the number of coding units in a video frame, R.sub.n is the allocated bits for the n.sup.th coding unit, A.sub.n represents the visual attention value of the n.sup.th coding unit in a video frame, f.sub.i is the visual attention value of a pixel, and denotes the set of blocks in a coding unit area, and N.sub.b is the number of pixels in the coding unit; and encoding the light field video based on the bit allocation scheme.

2. The method of claim 1, wherein the region of interest contains a human face.

3. The method of claim 2, wherein the region of interest contains a feature of a human face.

4. The method of claim 2, wherein the visual attention value is determined based on a saliency map of the human face.

5. The method of claim 4, wherein a subjective rate-distortion optimization is performed to allocate more resources to the region of interest.

6. The method of claim 1, further comprising: calculating an LCU level Lagrange multiplier in according to ${\lambda }_{new}={\left(\frac{A_n}{A_{\mathrm{avg}}}\right)}^{\beta }\mathrm{•\lambda },$ wherein A.sub.avg is the average visual attention value.

7. The method of claim 6, wherein β is set to −0.9.

8. The method of claim 6, wherein $\frac{A_n}{A_{\mathrm{avg}}}$ is set in a range from 0.5 to 2.

9. The method of claim 1, further comprising: calculating an LCU level quantization parameter in according to ${\mathrm{QP}}_{\mathrm{new}}=QP+3\mathrm{\beta •log}\left(\frac{A_n}{A_{\mathrm{avg}}}\right).$

10. A light field video system, comprising: a light field array system for capturing a plurality of videos and a plurality of depth maps to generate a light field video; a processer; and a memory that stores at least one program to control the light field video system, wherein the processor is configured to execute the program for: determining a region of interest in the light field video; assigning a visual attention value to the region of interest, wherein the visual attention value is different from that for other regions of the light field video; calculating a bit allocation scheme based on the visual attention value of the region of interest, wherein the bit allocation scheme is calculated in accordance to: $\begin{array}{c}{R}_n^{\prime }=\frac{A_n}{\underset{i=1}{\overset{N}{.Math.}}{A}_i}*R_{\mathrm{frame}}\\ A_n=\frac{\underset{i=1}{\overset{N_b}{.Math.}}{f}_i}{N_b}\end{array}$ wherein R.sub.frame is the total bits of a video frame, N is the number of coding units in a video frame, R.sub.n is the allocated bits for the n.sup.th coding unit, A.sub.n represents the visual attention value of the n.sup.th coding unit in a video frame, f.sub.i is the visual attention value of a pixel, and denotes the set of blocks in a coding unit area, and N.sub.b is the number of pixels in the coding unit; and encoding the light field video based on the bit allocation scheme.

11. The light field video system of claim 10, wherein the region of interest contains a human face.

12. The light field video system of claim 11, wherein the region of interest contains a feature of a human face.

13. The light field video system of claim 11, wherein the visual attention value is determined based on a saliency map of the human face.

14. The light field video system of claim 13, wherein a subjective rate-distortion optimization is performed to allocate more resources to the region of interest.

15. The light field video system of claim 10, wherein the processor is further configured to execute the program for: calculating an LCU level Lagrange multiplier in according to ${\lambda }_{new}={\left(\frac{A_n}{A_{\mathrm{avg}}}\right)}^{\beta }\mathrm{•\lambda },$ wherein A.sub.avg is the average visual attention value.

16. The light field video system of claim 15, wherein β is set to −0.9.

17. The light field video system of claim 15, wherein $\frac{A_n}{A_{\mathrm{avg}}}$ is set in a range from 0.5 to 2.

18. The light field video system of claim 10, wherein the processor is further configured to execute the program for: calculating an LCU level quantization parameter in according to ${\mathrm{QP}}_{\mathrm{new}}=QP+3\mathrm{\beta •log}\left(\frac{A_n}{A_{\mathrm{avg}}}\right).$

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows overall architecture of the perceptual rate-distortion optimization (RDO) scheme for omnidirectional video coding.

(2) FIG. 2A is a schematic showing of face region detection, wherein a region of interest contains a human face.

(3) FIG. 2B is an image containing one face produced by the light field video compression method in accordance with an embodiment of the present invention.

(4) FIG. 2C is an image containing one face produced in accordance with the HEVC compression method.

(5) FIG. 2D is a is a schematic showing of face region detection, wherein a region of interest contains two human faces.

(6) FIG. 2E is an image containing two faces produced by the light field video compression method in accordance with an embodiment of the present invention.

(7) FIG. 2F is an image containing two faces produced in accordance with the HEVC compression method.

DETAIL DESCRIPTION OF SOME EMBODIMENTS

(8) In this disclosure, a perceptual light field video coding approach based on face detecting is provided. Perceptual video coding is designed to reduce the perceived redundancy by enhancing the visual quality of the face region at the cost of the quality in other regions. Thus, it is an effective solution to transmit a large amount of video data in light field system. Specifically, a perceptual model of multiple faces for videos is provided. To this end, a method is developed to detect face and facial features of multiple faces in light field system. Then, different weights are assigned to different regions in a video frame with one or more faces. Based on such weights, a subjective rate-distortion optimization was performed to improve the quality of regions with important information, such as face and facial features. Experimental results showed that this approach produces better face recognition quality than HEVC.

(9) 1. Face Region Weighted Based LCU Level Rate Control Scheme

(10) The frames in a natural video cannot be considered as a stack of independent still images, since they also contain critical motion information related to these images. Moreover, perception of motion information between frames is an important feature for video quality assessment (VQA). In the conventional video coding framework such as HEVC, once the prediction residual is obtained, all the LCUs are considered equally for bit allocation, which leads to a constant lamda and QP at LCU level. This approach does not conform with HVS, because the perceptual information of LCUs in each frame is different, and the fixed lamda and QP can hardly result in optimum bits allocation. Thus, a novel bit-rate allocation strategy based on the perceptual model of faces is provided in accordance with embodiments of the present invention.

(11) In HEVC, the hyperbolic model is adopted to characterize the R-D relationship, which is expressed as

(12) $\begin{array}{cc}\lambda =-\frac{\partial D}{\partial R}=\mathrm{\alpha •}{R}^{\beta }& \left(1\right)\end{array}$

(13) where α and β are parameters. As we analyzed above, a constant lamda and QP at LCU level indicates that all the LCUs are considered equally for bit allocation after getting the prediction residual in the conventional video coding framework, which can be expressed as

(14) 0$\begin{array}{cc}{R}_n=\frac{R_{\mathrm{frame}}}{N}& \left(2\right)\end{array}$

(15) where R.sub.frame is the total bits of a frame, N is the number of LCUs in the frame, R.sub.n is the allocated bits for the n.sup.th LCU.

(16) However, the capability of HVS to detect distortions in the video sequences must be taken into account to develop a video coder with better face recognition quality. More bits should be allocated to the LCUs where human can easily see coding distortions, and fewer bits should be allocated to the LCUs where coding distortions are less noticeable. Since face attracts extensive human visual attention, the proposed bit allocation scheme is calculated based on saliency map of face as follow:

(17) $\begin{array}{cc}{R}_n^{\prime }=\frac{A_n}{\underset{i=1}{\overset{N}{.Math.}}{A}_i}•R_{\mathrm{frame}}& \left(3\right)\\ A_n=\frac{\underset{i=1}{\overset{N_b}{.Math.}}{f}_i}{N_b}& \end{array}$

(18) where A.sub.n represents the visual attention value of the n.sup.th LCU in a frame, f.sub.i is visual attention value of a pixel, denotes the set of blocks in a LCU area, and N.sub.b is the number of pixels in a LCU.

(19) 2. Adaptive LCU Level Lagrange Multiplier and Quantization Adjustment

(20) By substituting Equations (2) and (3) into Equation (1), a new Lagrange multiplier can be obtained as follow:

(21) $\begin{array}{cc}{\lambda }_{\mathrm{new}}={\left(\frac{A_n}{A_{\mathrm{avg}}}\right)}^{\beta }\mathrm{•\lambda }& \left(4\right)\end{array}$

(22) where A.sub.avg is the average visual attention value. In this embodiment, β is set to −0.9. To avoid the sudden change of Lagrange multiplier, the

(23) $\frac{A_n}{A_{\mathrm{avg}}}$
is clipped into [0.5,2].

(24) From Equation (4), we know that a LCU with smaller visual attention will have a larger λ.sub.new, while a LCU with larger visual attention will have a smaller λ.sub.new. As λ.sub.new is used to balance the distortion and rate, a smaller λ.sub.new brings a lower coding distortion with more coding bits, while a larger λ.sub.new results in a higher coding distortion with less coding bits. Thus, the proposed λ.sub.new does conform with HVS.

(25) In addition, since Lagrange multiplier is generally modeled as a function of quantization parameter as follow:
λ=0.85□2.sup.(QP−12)/3.0  (5)

(26) where QP is the quantization parameter. By combining Equations (4) and (5), the new quantization parameter can be obtained as follow:

(27) $\begin{array}{cc}{\mathrm{QP}}_{\mathrm{new}}=QP+3\mathrm{\beta •log}\left(\frac{A_n}{A_{\mathrm{avg}}}\right)& \left(6\right)\end{array}$

(28) As shown from Equation (6), the LCUs with less visual attention are quantized more coarsely when compared with the perceptually important regions. By doing so, the perceptually less important regions such as background are compromised to save bits for the face regions. Moreover, there are more choices for perceptual optimization as the QP could vary in a range. Thus, we can get better visual quality at the same bit rate.

(29) The overall architecture of the perceptual RDO scheme for omnidirectional video coding is exhibited in FIG. 1.

(30) The experimental results of the proposed method are shown in FIGS. 2A-2F.

(31) FIG. 2A is a schematic showing of face region detection, wherein a region of interest contains a human face.

(32) FIG. 2B is an image containing one face produced by the light field video compression method in accordance with an embodiment of the present invention.

(33) FIG. 2C is an image containing one face produced in accordance with the HEVC compression method.

(34) FIG. 2D is a is a schematic showing of face region detection, wherein a region of interest contains two human faces.

(35) FIG. 2E is an image containing two faces produced by the light field video compression method in accordance with an embodiment of the present invention.

(36) FIG. 2F is an image containing two faces produced in accordance with the HEVC compression method.

(37) As shown in FIG. 2A-2F, this method provides better visual quality with fewer artifacts and smaller blurring in the face regions. Thus, the visual quality and peak signal to noise ratio (PSNR) in the face regions have been significantly improved.