Establishment method of 3D Saliency Model Based on Prior Knowledge and Depth Weight

Abstract

A method of establishing a 3D saliency model based on 3D contrast and depth weight, includes dividing left view of 3D image pair into multiple regions by super-pixel segmentation method, synthesizing a set of features with color and disparity information to describe each region, and using color compactness as weight of disparity in region feature component, calculating feature contrast of a region to surrounding regions; obtaining background prior on depth of disparity map, and improving depth saliency through combining the background prior and the color compactness; taking Gaussian distance between the depth saliency and regions as weight of feature contrast, obtaining initial 3D saliency by adding the weight of the feature contrast; enhancing the initial 3D saliency by 2D saliency and central bias weight.

Claims

1. A method of establishing a 3D saliency model based on 3D contrast and depth weight including the following steps of: step one: extracting 3D feature: dividing left view of 3D image pair into N regions by super-pixel segmentation method, labeling as R.sub.i, where i takes value 1 to N, N is an integer; defining a region feature f=[l, a, b, d] for region R.sub.i, wherein l=.sub.i=1.sup.N.sup.il.sub.i/N.sub., a=.sub.i=1.sup.N.sup.ia.sub.i/N.sub.i, b=.sub.i=1.sup.N.sup.ib.sub.i/N.sub.i, $\underset{\_}{d=\underset{i=1}{\overset{N_i}{.Math.}}.Math..Math.d_i/N_i},$ N.sub.i is number of pixels in the region R.sub.i, and l.sub.i, a.sub.i, b.sub.id.sub.i is value of l, a, b and disparity of pixel in the region R.sub.i, respectively; step two: calculating feature contrast: representing feature contrast between regions by matrix C, then c.sub.ij represents norm distance between regional features of the region R.sub.i and region R.sub.j, which is calculated as: c.sub.ij=u.sub.if.sub.i-u.sub.jf.sub.j.sub.2, wherein u is weight of region feature f, u=[1,1,1,q], and variable q represents color compactness of N regions in the left view; step three: designing weight of the feature contrast: (1) obtaining depth saliency map s.sub.s by depth domain analysis method on disparity map, then depth saliency s.sub.d of the region R.sub.i is calculated by using a formula as: s.sub.d(i)=s.sub.s(i)e.sup.kt.sup.i; (2) calculating background prior on the disparity map; (3) optimizing depth saliency through the background prior, by using specific process including the step of: for the region R.sub.i, using mean disparity d.sub.i of the region R.sub.i on the disparity map to determine whether it appears to have no correlation with background range and the depth saliency s.sub.d(i) is within background range, and the depth saliency is determined by using a formula: $s_d\left(i\right)=\{\begin{array}{cc}{s}_d\left(i\right),& {\stackrel{\_}{d}}_i<\mathrm{thresh}\\ 0,& {\stackrel{\_}{d}}_i\mathrm{thresh}\end{array},$ wherein threshold thresh is minimum disparity of portion marked as background on the disparity map of depth background B.sub.d; (4) designing the weight of feature contrast: the weight of feature contrast of the region R.sub.i and the region R.sub.j represented by a variable w.sub.i,j, and there are:
w.sub.i,j=exp (Dst(i,j)/.sup.2)a(i)s.sub.d(i), wherein (i) is size of the region R.sub.i, exp (Dst(i, j)/.sup.2) represents Gaussian distance between the region R.sub.i and the region R.sub.j; step four: calculating initial 3D saliency: saliency value of the region R.sub.i is s.sup.r(i)=e.sup.kt.sup.i.sub.ijw.sub.i,jc.sub.i,j, then calculation formula of the initial 3D saliency s.sup.p(i) of the region R.sub.i is
S.sup.p(i)=.sub.j=1.sup.N.sup.iexp ((clr.sub.i-clr.sub.j.sup.2+p.sub.i-p.sub.j.sup.2))S.sup.r(j), wherein =0.33, =0.33, are two parameters to control the sensitivity color distance (clr.sub.i-clr.sub.j) and position distance (p.sub.i-p.sub.j), respectively, N.sub.i is the number of pixels in the region R.sub.i; step five: enhancing initial 3D saliency: final 3D saliency s(i) of the region R.sub.i is s(i)=CBW(i)*s.sub.pca.sup.r(i)*s.sup.p(i), wherein S.sub.pca.sup.r (i) is 2D saliency of the region R.sub.i, S.sub.pca.sup.r(i)=.sub.pr.sub.iS.sub.pca(p)/N.sub.i, S.sub.pca(p) is saliency at pixel level, $\mathrm{CBW}\left(i\right)=\{\begin{array}{c}0,.Math.p_{i}B,\\ \exp (-\left(\mathrm{DstToCt}\left(i\right)\right)/\left(2.Math.{}_{\mathrm{xy}}^2\right),\end{array}$ wherein DstToCt(i) is Euclidean distance for pixel to center coordinate, B=(B.sub.bB.sub.d), .sub.xy={square root over (H*H+W*W)}/2, H and W are width and height of the left view, B.sub.d represents depth background, and B.sub.b represents boundary background.

2. The method according to claim 1, being characterized in that: in said step two, q=e.sup.kt.sup.i, k is Gaussian scale factor, k=4, t.sub.i is calculated as: t.sub.i=.sub.j=1.sup.Np.sub.j.sub.i.sup.2dis.sub.ij.sup.clr.sup.i, dis.sub.ij.sup.clr.sup.i is color distance of RGB mean of the region R.sub.i and the region R.sub.j, ${\mathrm{dis}}_{\mathrm{ij}}^{\mathrm{clr}}=\exp \left(\frac{1}{2.Math.{}_c^2}.Math.{.Math.{\mathrm{clr}}_i-{\mathrm{clr}}_j.Math.}^2\right).Math.p_j$ is center coordinate of centroid of the region R.sub.j, and .sub.i is weight position of color clr.sub.i, .sub.i.sub.j=1.sup.Ndis.sub.ij.sup.clrp.sub.j.

3. The method according to claim 1, being characterized in that: in said step (2), specific process for calculating background prior on disparity map includes the steps of: (a) defining initial background image: B.sub.d=0; (b) initializing the furthest background, first, finding coordinate of the largest disparity in disparity map I.sub.d, P.sub.xy=Pos(max(I.sub.d)); then setting initial value (P.sub.xy)=1; (c) calculating background propagation: B.sub.d=Contour ((P.sub.xy)), wherein symbol Contour represents segmentation based on active contour, pixel of background portion in the depth background B.sub.d is denoted as 1, and pixel of foreground portion are represented as 0.

4. The method according to claim 2, being characterized in that: in said step (2), specific process for calculating background prior on disparity map includes the steps of: (a) defining initial background image: B.sub.d=0; (b) initializing the furthest background, first, finding the coordinate of the largest disparity in disparity map I.sub.d, P.sub.xy=Pos(max(I.sub.d)); then setting initial value (P.sub.xy)=1; (c) calculating background propagation: B.sub.d=Contour((P.sub.xy)), wherein symbol Contour represents segmentation based on active contour, the pixel of background portion in the depth background image B.sub.d is denoted as 1, and the pixel of foreground portion are represented as 0.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG. 1 is a flow diagram of the inventive establishment method of 3D saliency model based on 3D contrast and depth weight;

[0043] FIG. 2a is a display diagram of ROC (Receiver operating feature) curve performance, wherein abscissa is False Positive Rate(FPR), ordinate is True Positive Rate(TPR).

[0044] FIG. 2b is PR(Precision-Recall) curve, wherein abscissa is recall rate. Ordinate is predicted precision. Icon DWRC (depth-weighted region contrast) is abbreviation of the present invention method in the FIG. 2a and FIG. 2b;

[0045] FIG. 3a is a left side view of the 3D image pair in one embodiment of the present invention. FIG. 3b is a right side view of the 3D image pair;

[0046] FIG. 3b is a right side view of the 3D image pair in one embodiment of the present invention;

[0047] FIG. 3c is a disparity map in one embodiment of the present invention;

[0048] FIG. 3d is an initial 3D saliency map in one embodiment of the present invention;

[0049] FIG. 3e is a target graph (i.e., final 3D saliency map) in one embodiment of the present invention.

[0050] FIG. 4a is a left side view of the 3D image pair. FIG. 4b is a right side view of the 3D image pair in another embodiment of the present invention.

[0051] FIG. 4b is a right side view of the 3D image pair in another embodiment of the present invention.

[0052] FIG. 4c is a disparity map in another embodiment of the present invention.

[0053] FIG. 4d is an initial 3D saliency map in another embodiment of the present invention.

[0054] FIG. 4e is a target graph (i.e., final 3D saliency map) in another embodiment of the present invention.

DETAILED DESCRIPTION

[0055] The present invention will now be described in further detail with reference to the accompanying drawings as required:

[0056] As shown in FIG. 1, the present invention provides a method of establishing a 3D saliency model based on 3D contrast and depth weight, including: dividing left view of 3D image pair into multiple regions by super-pixel segmentation method, synthesizing a set of features with color and disparity information to describe each region, and using color compactness as weight of disparity in region feature component, calculating feature contrast of a region to surrounding regions; obtaining background prior on depth of disparity map, and improving depth saliency through combining the background prior and the color compactness; taking Gaussian distance between the depth saliency and regions as weight of feature contrast, obtaining initial 3D saliency by adding the weight of the feature contrast; enhancing the initial 3D saliency by 2D saliency and central bias weight. The 3D saliency model established by the invention has closer to the human gazing effect.

[0057] Specifically, the method of establishing a 3D saliency model of the present invention includes the steps of:

[0058] Step one: extracting 3D feature:

[0059] Dividing left view of 3D image pair into N regions by super-pixel segmentation method (SLIC), labeling as R.sub.i, where i takes value 1 to N; defining a region feature using CIELab color and disparity, namely, defining a region feature f=[l, a, b, d] for R.sub.i (features of discriminant R.sub.i are expressed as L*a*b mean and disparity mean of the color image in this region), wherein l=.sub.i=1.sup.N.sup.il.sub.i/N.sub.i, a=.sub.i=1.sup.N.sup.ia.sub.i/N.sub.i, b=.sub.i=1.sup.N.sup.ib.sub.i/N.sub.i,

[00005]$\underset{\_}{d=\underset{i=1}{\overset{N_i}{.Math.}}.Math..Math.d_i/N_i},$

N.sub.i is the number of pixels in the region R.sub.i, and l.sub.i, a.sub.i, b.sub.i, d.sub.i is value of l, a b and disparity of pixel in the region R.sub.i, respectively;

[0060] Step two: calculating feature contrast:

[0061] Representing feature contrast between regions by matrix C, then c.sub.ij represents norm distance between regional features of the region R.sub.i and the region R.sub.j, which is calculated as:

c.sub.ij=u.sub.if.sub.iu.sub.jf.sub.j.sub.2,

[0062] Wherein u is the weight of region feature f, u=[1,1,1,q],

[0063] Variable q represents color compactness of N regions in the left view, and is used to indicate distribution of colors of each region in the left view, q=e.sup.kt.sup.i, wherein k is Gaussian scale factor. k=4, t.sub.i is calculated as t.sub.i=.sub.j=1.sup.Np.sub.j.sub.i.sup.2dis.sub.ij.sup.clr.sup.i.

[0064] Wherein dis.sub.ij.sup.clr.sup.i is color distance of RGB mean of the region R.sub.i and the region R.sub.j,

[00006]${\mathrm{dis}}_{\mathrm{ij}}^{\mathrm{clr}}=\exp \left(\frac{1}{2.Math.{}_c^2}.Math.{.Math.{\mathrm{clr}}_i-{\mathrm{clr}}_j.Math.}^2\right).$

p.sub.j is center coordinate of centroid of the region R.sub.j, and .sub.i is weight position of color cir.sub.i, .sub.i=.sub.j=1.sup.Ndis.sub.ij.sup.clrp.sub.j.

[0065] Step three: designing weight of feature contrast:

[0066] After calculating the feature contrast C of each region, the weight of the feature contrast is represented by matrix W. w.sub.ij represents corresponding weight of C.sub.ij.

[0067] The weight of the feature contrast takes into account depth saliency, Gaussian distance exp (Dst(i,j)/.sup.2) between region size a(i) and regions. Wherein calculation process of the depth saliency s.sub.d is: obtaining result s.sub.I through domain analysis on the disparity map, and then using background prior (including depth background B.sub.d and boundary background B.sub.d) and color tightness (formula e.sup.kt.sup.i) to improve. The detailed process is as follows:

[0068] (1) calculating depth saliency map

[0069] Obtaining the depth saliency map s.sub.s by depth domain analysis method on the disparity map, obtaining s.sub.d through color tightness enhancement, then depth saliency s.sub.d of the region R.sub.i is calculated as: s.sub.d(i)=s.sub.s(i)e.sup.kt.sup.i;

[0070] 2) calculating background prior on the disparity map:

[0071] There are two stages to extract the background prior on the disparity map: background initialization and background propagation. Specific steps include:

[0072] (a) defining initial background image: B.sub.d=0;

[0073] (b) initializing the furthest background, first, finding coordinate of the largest disparity in disparity map I.sub.d, P.sub.xy=Pos(max(I.sub.d)); then setting initial value (P.sub.xy)=1;

[0074] (c) calculating background propagation: B.sub.d=Contour ((P.sub.xy)), , wherein symbol Contour represents segmentation based on active contour, pixel of background portion in the depth background B.sub.d is denoted as 1, and pixel of foreground portion are represented as 0.

[0075] (3) optimizing depth saliency through the background prior, specific process includes the step of:

[0076] For the region R.sub.i, using mean disparity d.sub.i of the region R.sub.i on the disparity map to determine whether it appears to have no correlation with background range and the depth saliency s.sub.d(i) is not within background range, the depth saliency is determined by using a formula:

[00007]$s_d\left(i\right)=\{\begin{array}{cc}{s}_d\left(i\right),& {\stackrel{\_}{d}}_i<\mathrm{thresh}\\ 0,& {\stackrel{\_}{d}}_i\mathrm{thresh}\end{array},$

[0077] Wherein threshold thresh is the minimum disparity of portion marked as background on the disparity map of the depth background B.sub.d namely, thresh=min(I.sub.d(q)), q {B.sub.d<0}.

[0078] The boundary background is B.sub.b. Background area in the boundary background is represented by 1, and the other areas are represented by 0. If the R.sub.i region is at the position of the boundary background, the saliency s.sub.d(i) is marked as 0, otherwise it is not changed.

[0079] (4) designing weight of feature contrast:

[0080] Weight of feature contrast of the region R.sub.i and the region R.sub.j is represented by a variable w.sub.ij. There are:

w.sub.ij=exp (Dst(i, j)/.sup.2)a(i)s.sub.d(i),

[0081] Wherein (i) is the size of the region R.sub.i, exp (Dst(i,j)/.sup.*) represents Gaussian distance between the region R.sub.i and the region R.sub.j.

[0082] Step four: calculating initial 3D saliency:

[0083] After completing calculation of the feature contrast c.sub.i,j and the weight w.sub.i,j of the region R.sub.i, saliency value of the region R.sub.i, can be calculated by the following formula:

S.sup.r(i)=e.sup.kt.sup.i.sub.i=jw.sub.ijc.sub.ij,

[0084] In order to eliminate effect of super-pixel segmentation errors, saliency (i.e., initial 3D significance) of super-pixel of each region is obtained by saliency linear combination of its surrounding regions. Saliency of super-pixel of the region R.sub.i calculation formula is:

S.sup.p(i)=.sub.j=1.sup.V.sup.iexp ((clr.sub.i-clr.sub.j.sup.2+p.sub.i-p.sub.j.sup.2))S.sup.r(j),

[0085] Wherein and are respectively parameters of control color distance (|clr.sub.i-clr.sub.j) and position distance (p.sub.i-p.sub.j), =0.33, =0.33 , are two parameters to control the sensitivity color distance (clr.sub.i-clr.sub.j) and position distance (p.sub.i-p.sub.j), respectively, N.sub.i is the number of pixels in the region R.sub.i.

[0086] Step five: enhancing the initial 3D saliency:

[0087] After calculating the initial 3D saliency S.sup.p(i), performing enhancement through 2D saliency and central bias weight. Final 3D saliency of super-pixel of the region R.sub.i is:

s(i)=CBW(i)*S.sub.pca.sup.r(i)*S.sup.p(i),

[0088] Wherein S.sub.pca.sup.r(i) is 2D saliency of the region R.sub.i, S.sub.pca.sup.r(l)=.sub.penS.sub.pca(p)/N.sub.i, S.sub.pca(p) is saliency at pixel level. CBW(i) (central bias weight) is a Gaussian function modified with background prior, and is calculated by the following formula:

[00008]$\mathrm{CBW}\left(i\right)=\{\begin{array}{c}0,.Math.p_{i}B,\\ \exp (-\left(\mathrm{DstToCt}\left(i\right)\right)/\left(2.Math.{}_{\mathrm{xy}}^2\right),\end{array}$

[0089] Wherein DstToCt(i) is Euclidean distance for pixel to center coordinate, B=(B.sub.bB.sub.d), .sub.xy={square root over (H*H+W*W)}/2. H and W are the width and height of the left view. B.sub.d represents depth background. B.sub.b represents boundary background.

[0090] Referring to FIG. 2a and FIG. 2b, near upper left corner point in curve of the FIG. 2a, AUC (area under roc curve) value is 0.89 calculated according to showing result in FIG. 2a; recall rate increase in FIG. 2b do not cause sharp decrease of accuracy, and F.sub.(0.3)=0.61 calculated as shown in FIG. 2b. That is, the present invention can obtain a 3D saliency model close to the human eye gaze.

[0091] Referring to FIG. 3a to FIG. 3e, FIG. 4a to FIG. 4e, in both embodiments, the establishment method of 3D saliency model according to the present invention is used to obtain 3D significance model close to the human eye gaze. Seeing from FIG. 3e and FIG. 4e, regions of color contrast and disparity contrast have high saliency values, and background interference is eliminated, then target's saliency is improved.

[0092] In the present inventive method, features are taken from color image and disparity map, and the feature contrast is calculated by using color compactness. In addition to conventional boundary background prior, also using background prior extracted from the disparity map according to the distance from object to observer, and object compactness in color image as supplement of depth saliency, depth saliency of the disparity map is taken as weight of the feature contrast to obtain the initial 3D saliency. Then, performing enhancement for initial 3D saliency through using 2D saliency and the central bias weight. Because feature is not only from the color information of the 2D image, but also contains information of the depth channel, in combination with prior knowledge such as background and color compactness, the 3D saliency model of the present invention has closer to the human gazing effect.

[0093] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications previously mentioned in the specification and embodiments, and can be applied in various fields suitable for the present invention. For ordinary skilled person in the field, other various changed model, formula and parameter may be easily achieved without creative work according to instruction of the present invention, changed, modified and replaced embodiments without departing the general concept defined by the claims and their equivalent are still included in the present invention. The present invention is not limited to particular details and illustrations shown and described herein.