Video quality objective assessment method based on spatiotemporal domain structure
09756323 · 2017-09-05
Assignee
Inventors
Cpc classification
H04N19/89
ELECTRICITY
International classification
H04N19/154
ELECTRICITY
Abstract
A video quality objective assessment method based on a spatiotemporal domain structure firstly combines a spatiotemporal domain gradient magnitude and color information for calculating a spatiotemporal domain local similarity, and then uses variance fusion for spatial domain fusion. The spatiotemporal domain local similarity is fused into frame-level objective quality value, and then a temporal domain fusion model is established by simulating three important global temporal effects, which are a smoothing effect, an asymmetric track effects and a recency effect, of a human visual system. Finally, the objective quality values of the distorted video sequence are obtained. By modeling the human visual temporal domain effect, the temporal domain weighting method of the present invention is able to accurately and efficiently evaluate the objective quality of the distorted video.
Claims
1. A video quality objective assessment method based on a spatiotemporal domain structure, comprising steps of: (1) marking a reference video sequence without distortion as S.sub.r, and marking a distorted video sequence, which is obtained after the S.sub.r is distorted, as S.sub.d; wherein a total S.sub.r frame quantity is F, a total S.sub.d frame quantity is also F, and F>1; widths of images in both the S.sub.r and the S.sub.d are W, and heights of the images in both the S.sub.r and the S.sub.d are H; defining an image luminance component sequence of the images in the S.sub.r as a luminance component sequence of the S.sub.r and marking as Y.sub.r; defining a first image chrominance component sequence of the images in the S.sub.r as a first chrominance component sequence of the S.sub.r and marking as U.sub.r; defining a second image chrominance component sequence of the images in the S.sub.r as a second chrominance component sequence of the S.sub.r and marking as V.sub.r; defining an image luminance component sequence of the images in the S.sub.d as a luminance component sequence of the S.sub.d and marking as Y.sub.d; defining a first image chrominance component sequence of the images in the S.sub.d as a first chrominance component sequence of the S.sub.d and marking as U.sub.d; defining a second image chrominance component sequence of the images in the S.sub.d as a second chrominance component sequence of the S.sub.d and marking as V.sub.d; wherein widths of images in the Y.sub.r, the U.sub.r, the V.sub.r, the Y.sub.d, the U.sub.d and the V.sub.d are W, and heights of the images in the Y.sub.r, the U.sub.r, the V.sub.r, the Y.sub.d, the U.sub.d and the V.sub.d are H; (2) calculating a spatiotemporal domain gradient magnitude sequence of the Y.sub.r with a three-dimensional Prewitt operator and marking as G.sub.r, and marking a pixel value of a pixel at a position of (x,y) in a number t frame in the G.sub.r as G.sub.r(x,y,t), wherein
2. The video quality objective assessment method, as recited in claim 1, wherein in the step (3), λ=3.
3. The video quality objective assessment method, as recited in claim 1, wherein in the step (7), α=0.03, β=0.2, γ=1000.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
(3) Referring to drawings and a preferred embodiment, the present invention is further illustrated.
(4) Outstanding full-reference video quality assessment methods not only are highly accurate in prediction and sensitive to both spatial and temporal distortions, but also lower a computational complexity as much as possible and provide real-time processing of video sequences. The present invention combines a spatiotemporal domain gradient magnitude and color information for calculating a spatiotemporal domain local similarity, and then treats a whole video sequence as a dynamic temporal sequence. Firstly, frame-level objective quality value of each frame is obtained by variance fusion, then a temporal domain fusion model is established by simulating three important global temporal effects, which are a smoothing effect, an asymmetric track effects and a recency effect, of a human visual system, so as to obtain objective quality values of a distorted video sequence. Bottom feature calculation is simple and sensitive to both temporal and spatial distortions, and the temporal domain fusion model analogs a temporal domain effect for ensuring accuracy and efficiency of the present invention.
(5) The present invention provides a video quality objective assessment method based on a spatiotemporal domain structure, whose overall flow chart is shown in
(6) (1) marking a reference video sequence without distortion as S.sub.r, and marking a distorted video sequence, which is obtained after the S.sub.r is distorted, as S.sub.d; wherein a total S.sub.r frame quantity is F, a total S.sub.d frame quantity is also F, and F>1; widths of images in both the S.sub.r and the S.sub.d are W, and heights of the images in both the S.sub.r and the S.sub.d are H; defining an image luminance component sequence of the images in the S.sub.r as a luminance component sequence of the S.sub.r and marking as Y.sub.r; defining a first image chrominance component sequence of the images in the S.sub.r as a first chrominance component sequence of the S.sub.r and marking as U.sub.r; defining a second image chrominance component sequence of the images in the S.sub.r as a second chrominance component sequence of the S.sub.r and marking as V.sub.r; defining an image luminance component sequence of the images in the S.sub.d as a luminance component sequence of the S.sub.d and marking as Y.sub.d; defining a first image chrominance component sequence of the images in the S.sub.d as a first chrominance component sequence of the S.sub.d and marking as U.sub.d; defining a second image chrominance component sequence of the images in the S.sub.d as a second chrominance component sequence of the S.sub.d and marking as V.sub.d; wherein widths of images in the Y.sub.r, the U.sub.r, the V.sub.r, the Y.sub.d, the U.sub.d and the V.sub.d are W, and heights of the images in the Y.sub.r, the U.sub.r, the V.sub.r, the Y.sub.d, the U.sub.d and the V.sub.d are H;
(7) (2) calculating a spatiotemporal domain gradient magnitude sequence of the Y.sub.r with a three-dimensional Prewitt operator and marking as G.sub.r, and marking a pixel value of a pixel at a position of (x,y) in a number t frame in the G.sub.r as G.sub.r(x,y,t), wherein
(8)
Y.sub.rx=Y.sub.r{circle around (×)}F.sub.x, Y.sup.ry=Y.sub.r{circle around (×)}F.sub.y, Y.sub.rt=Y.sub.r{circle around (×)}F.sub.t;
(9) similarly, calculating a spatiotemporal domain gradient magnitude sequence of the Y.sub.d with the three-dimensional Prewitt operator and marking as G.sub.d, and marking a pixel value of a pixel at a position of (x,y) in a number t frame in the G.sub.d as G.sub.d(x,y,t), wherein
(10)
Y.sub.dx=Y.sub.d{circle around (×)}F.sub.x, Y.sub.dy=Y.sub.d{circle around (×)}F.sub.y, Y.sub.dt=Y.sub.d{circle around (×)}F.sub.t;
(11) wherein an initial value of the t is 1, 1≦t≦F, 1≦x≦W, 1≦y≦H; Y.sub.rx(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a horizontal gradient magnitude sequence Y.sub.rx of the Y.sub.r, Y.sub.ry(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a vertical gradient magnitude sequence Y.sub.ry of the Y.sub.r, and Y.sub.rt(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a temporal gradient magnitude sequence Y.sub.rt of the Y.sub.r; Y.sub.dx(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a horizontal gradient magnitude sequence Y.sub.dx of the Y.sub.d, Y.sub.dy(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a vertical gradient magnitude sequence Y.sub.dy of the Y.sub.d, and Y.sub.dt(x,y,t) refers to a pixel value of a pixel at a position of (x,y) in a number t frame in a temporal gradient magnitude sequence Y.sub.dt of the Y.sub.d; a symbol {circumflex over (×)} is a zeros truncated convolution symbol; after convolution, dimensions of the Y.sub.rx, the Y.sub.ry and the Y.sub.rt are same as a dimension of the Y.sub.r, and dimensions of the Y.sub.dx, Y.sub.dy and Y.sub.dt are same as a dimension of the Y.sub.d; F.sub.x, F.sub.y and F.sub.t correspond to a horizontal mask, a vertical mask and a temporal mask of the three-dimensional Prewitt operator; the F.sub.x, the F.sub.y and the F.sub.t are shown in
(12) (3) calculating a spatiotemporal domain local gradient similarity between each pixel point in each frame in the S.sub.r and a corresponding pixel point in a corresponding frame in the S.sub.d; marking the spatiotemporal domain local gradient similarity between a pixel point at a position of (x,y) in a number t frame in the S.sub.r and a pixel point at a position of (x,y) in a number t frame in the S.sub.d as G.sub.sim(x,y,t); wherein
(13)
c.sub.1 is a positive constant preventing the fractional from being meaningless; according the preferred embodiment, c.sub.1=90;
(14) (4) calculating a spatiotemporal domain local color similarity between each pixel point in each frame in the S.sub.r and the corresponding pixel point in the corresponding frame in the S.sub.d; marking the spatiotemporal domain local color similarity between the pixel point at the position of (x,y) in the number t frame in the S.sub.r and the pixel point at the position of (x,y) in the number t frame in the S.sub.d as C.sub.sim(x,y,t); wherein
(15)
U.sub.r(x,y,t) refers to a pixel value of a pixel point at a position of (x,y) in a number t frame in the U.sub.r, which is also a pixel value of a pixel point at a position of (x,y) in a first chrominance component in a number t frame in the S.sub.r; V.sub.r(x,y,t) refers to a pixel value of a pixel point at a position of (x,y) in a number t frame in the V.sub.r, which is also a pixel value of a pixel point at a position of (x,y) in a second chrominance component in a number t frame in the S.sub.r; U.sub.d(x,y,t) refers to a pixel value of a pixel point at a position of (x,y) in a number t frame in the U.sub.d, which is also a pixel value of a pixel point at a position of (x,y) in a first chrominance component in a number t frame in the S.sub.d; V.sub.d(x,y,t) refers to a pixel value of a pixel point at a position of (x,y) in a number t frame in the V.sub.d, which is also a pixel value of a pixel point at a position of (x,y) in a second chrominance component in a number t frame in the S.sub.d; c.sub.2 and c.sub.3 are positive constants preventing the fractional from being meaningless; according to the preferred embodiment, c.sub.2=c.sub.3=300;
(16) (5) calculating a spatiotemporal domain local similarity between each pixel point in each frame in the S.sub.r and the corresponding pixel point in the corresponding frame in the S.sub.d according to the spatiotemporal domain local gradient similarity between each pixel point in each frame in the S.sub.r and the corresponding pixel point in the corresponding frame in the S.sub.d, and the spatiotemporal domain local color similarity between each pixel point in each frame in the S.sub.r and the corresponding pixel point in the corresponding frame in the S.sub.d; marking the spatiotemporal domain local similarity between the pixel point at the position of (x,y) in the number t frame in the S.sub.r and the pixel point at the position of (x,y) in the number t frame in the S.sub.d as Q.sub.LS(x,y,t), wherein Q.sub.LS(x,y,t)=G.sub.sim(x,y,t)×(C.sub.sim(x,y,t)).sup.λ, λ is used for adjusting weights of color components, λ>0; according to the preferred embodiment, λ=3;
(17) (6) calculating an objective quality value of each frame in the S.sub.d with a variance fusion method, and marking the objective quality value of the number t frame in the S.sub.d as Q.sub.frame(t), wherein
(18)
Q.sub.mean(t) refers to an average value of the spatiotemporal domain local similarity between all pixel points in the number t frame in the S.sub.r and all pixel points in the number t frame in the S.sub.d,
(19)
and
(20) (7) calculating an objective quality value of the S.sub.d with a temporal domain weighting method and marking as Q, wherein
(21)
ΔQ.sub.frame(t)=Q.sub.frame(t)−Q.sub.LP(t−1), α refers to a weight during quality increase and β refers to a weight during quality decrease, γ is used to adjust a strength of a recency effect; according to the preferred embodiment, α=0.03, β=0.2, γ=1000.
(22) For illustrating feasibility and effectiveness, the present invention is tested as follows.
(23) A LIVE video database and a CSIQ video database are used. Referring to the LIVE video database, there are 10 video segments without distortion; four distortion types are involved, which are MPEG-2 compression distortion, H.264 compression distortion, transmission distortion of bit-stream compressed with H.264 standard through IP network, and transmission distortion through wireless network; there are totally 150 segments of distorted videos which are all YUV420 format with a resolution of 768×432 and a length of 10 seconds; and two frame rates, 25 frames per second and 50 frames per second, are used. Referring to the CSIQ video database, there are 12 video segments without distortion; six distortion types are involved, which are motion JPEG compression distortion, H.264 compression distortion, HEVC compression distortion, wavelet compression distortion, loss distortion of wireless transmission, and additive white Gaussian noise distortion; there are totally 216 segments of distorted videos which are all YUV420 format with a resolution of 832×480 and a length of 10 seconds; and five frame rates, 24 frames per second, 25 frames per second, 30 frames per second, 50 frames per second and 60 frames per second, are used. Both the LIVE video database and the CSIQ video database provide average subjective opinion scores of each distorted video. Since two chrominance components are both ¼ of a luminance component in the YUV420 format, the luminance component is processed with 2×2 mean filter in the spatial domain and down-sampling by a factor of 2, so as to matching dimensions of luminance component and chrominance components. Then the steps (1) to (7) are executed, and objective quality values of all distorted videos are calculated in a same way. For other formats such as YUV444 and YUYV, since the present invention requires that the dimension of the luminance component equal to the dimensions of the first chrominance component and the second chrominance component, down-sampling or up-sampling is also needed, so as to matching dimensions of the luminance component and the first as well as the second chrominance components. According to the present invention, in each treated video, sizes of all frames are equal.
(24) Three common objective parameters of the video quality assessment method are used as evaluation criteria, which are Pearson linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC) and rooted mean squared error (RMSE). Valve ranges of PLCC and SROCC are [0,1], wherein if a value is closer to 1, the assessment method is better; otherwise, the assessment method is worse. If a RMSE value is smaller, the assessment method is better; otherwise, the assessment method is worse. The PLCC, the SROCC and the RMSE values indicating assessment performances of the LIVE video database are shown in a Table 1, and the PLCC, the SROCC and the RMSE values indicating assessment performances of the CSIQ video database are shown in a Table 2. Referring to Table 1, values of the PLCC and the SROCC are higher than 0.84. Referring to Table 2, values of the PLCC and the SROCC are higher than 0.80. That is to say, with the present invention, the objective quality values calculated is highly related to the average subjective opinion scores, indicating that objective assessment results are relatively consistent with subjective perception of human eyes, which illustrates the effectiveness the present invention.
(25) TABLE-US-00001 TABLE 1 relativity between objective quality values of distorted videos obtained by the present invention and average subjective opinion scores for LIVE video database transmission trans- distortion of mission MPEG-2 H.264 H.264 compres- distortion all compres- compres- sed bit flow through dis- sion dis- sion dis- through IP wireless torted tortion tortion network network videos PLCC 0.9115 0.8260 0.8401 0.8816 0.8632 SROCC 0.9122 0.7660 0.8126 0.8332 0.8475 RMSE 4.6162 6.3169 5.4474 5.3174 5.5439
(26) TABLE-US-00002 TABLE 2 relativity between objective quality values of distorted videos obtained by the present invention and average subjective opinion scores for CSIQ video database additive motion loss white JPEG H.264 HEVC wavelet distortion of Gaussian all compression compression compression compression wireless noise distorted distortion distortion distortion distortion transmission distortion videos PLCC 0.9366 0.7976 0.8590 0.8748 0.9030 0.9586 0.8099 SROCC 0.9331 0.7985 0.8474 0.8440 0.8723 0.9416 0.8302 RMSE 6.9568 6.3461 7.6241 7.3420 4.6853 6.4334 9.7535
(27) One skilled in the art will understand that the embodiment of the present invention as shown in the drawings and described above is exemplary only and not intended to be limiting.
(28) It will thus be seen that the objects of the present invention have been fully and effectively accomplished. Its embodiments have been shown and described for the purposes of illustrating the functional and structural principles of the present invention and is subject to change without departure from such principles. Therefore, this invention includes all modifications encompassed within the spirit and scope of the following claims.