Method and apparatus for detecting salient object in image

Abstract

A method and an apparatus for detecting a salient object in an image includes separately performing convolution processing corresponding to at least two convolutional layers on a to-be-processed image to obtain at least two first feature maps of the to-be-processed image, performing superposition processing on at least two first feature maps included in a superposition set in at least two sets to obtain at least two second feature maps of the to-be-processed image, the at least two sets are in a one-to-one correspondence with the at least two second feature maps, and a resolution of a first feature map included in the superposition set is lower than or equal to a resolution of a second feature map corresponding to the superposition set, and splicing the at least two second feature maps to obtain a saliency map.

Claims

1. A method for detecting a salient object in an image, comprising: separately performing a first convolution processing corresponding to at least two convolutional layers on the image to obtain at least two first feature maps of the image, wherein resolutions of the at least two first feature maps are lower than a resolution of the image, and wherein a resolution of each of the at least two first feature maps is different; processing the at least two first feature maps to obtain at least two second feature maps of the image, wherein the at least two second feature maps are obtained by performing a first superposition processing on a part or all of the at least two first feature maps, wherein a resolution of each of the at least two second feature maps is different, and wherein the resolution of each of the at least two second feature maps is higher than or equal to a maximum resolution in the part or all of the at least two first feature maps; and splicing the at least two second feature maps based on a third weight corresponding to each of the at least two second feature maps to obtain a saliency map of the image, wherein the third weight is based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

2. The method of claim 1, wherein performing the first superposition processing on the part or all of the at least two first feature maps comprises: upsampling a first feature map of the at least two first feature maps, in the part or all of the at least two first feature maps, to obtain a third feature map corresponding to the first feature map, wherein a resolution of the first feature map is lower than the resolution of the at least one second feature map, and wherein a resolution of the third feature map is equal to the resolution of the at least one second feature map; and performing a second superposition processing on the third feature map and a second first feature map, in the part or all of the at least two first feature maps and on which upsampling is not performed, to obtain the at least one second feature map.

3. The method of claim 2, wherein performing the second superposition processing on the third feature map and the second first feature map comprises: obtaining a first weight corresponding to the second first feature map and a second weight corresponding to the third feature map; and performing, based on the first weight or the second weight, a third superposition processing on the third feature map and the second first feature map to obtain the at least one second feature map.

4. The method of claim 3, wherein the first weight or the second weight is obtained by training based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

5. The method of claim 2, wherein performing the second superposition processing on the third feature map and the second first feature map comprises performing superposition, convolution, and pooling processing on the third feature map and the second first feature map to obtain the at least one second feature map.

6. The method of claim 1, wherein splicing the at least two second feature maps comprises: performing a second convolution processing on the at least two second feature maps to obtain features of the at least two second feature maps; and splicing the features to obtain the saliency map of the image.

7. The method of claim 1, further comprising performing a first guided filtering on the saliency map of the image based on the image to obtain a segmented image of the image.

8. The method of claim 7, wherein the saliency map is a first saliency map, wherein a resolution of the first saliency map is lower than the resolution of the image, and wherein performing the first guided filtering on the saliency map of the image comprises: upsampling the first saliency map to obtain a second saliency map, wherein a resolution of the second saliency map is the same as the resolution of the image; and performing a second guided filtering on the second saliency map based on the image to obtain the segmented image.

9. An apparatus for detecting a salient object in an image, comprising: a memory comprising instructions; and a processor coupled to the memory, wherein the instructions cause the processor to be configured to: separately perform a first convolution processing corresponding to at least two convolutional layers on the image to obtain at least two first feature maps of the image, wherein resolutions of the at least two first feature maps are lower than a resolution of the image, and wherein a resolution of each of the at least two first feature maps is different; process the at least two first feature maps to obtain at least two second feature maps of the image, wherein the at least two second feature maps are obtained by performing a first superposition processing on a part or all of the at least two first feature maps, wherein a resolution of each of the at least two second feature maps is different, and wherein the resolution of each of the at least two second feature maps is higher than or equal to a maximum resolution in the part or all of the at least two first feature maps; and splice the at least two second feature maps based on a third weight corresponding to each of the at least two second feature maps to obtain a saliency map of the image, wherein the third weight is based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

10. The apparatus of claim 9, wherein the instructions further cause the processor to be configured to: upsample a first feature map of the at least two first feature maps, in the part or all of the at least two first feature maps, to obtain a third feature map corresponding to the first feature map, wherein a resolution of the first feature map is lower than the resolution of the at least one second feature map, and wherein a resolution of the third feature map is equal to the resolution of the at least one second feature map; and perform a second superposition processing on the third feature map and a second first feature map, in the part or all of the at least two first feature maps and on which upsampling is not performed, to obtain the at least one second feature map.

11. The apparatus of claim 10, wherein the instructions further cause the processor to be configured to: obtain a first weight corresponding to the second first feature map and a second weight corresponding to the third feature map; and perform, based on the first weight or the second weight, a third superposition processing on the third feature map and the second first feature map to obtain the at least one second feature map.

12. The apparatus of claim 11, wherein the first weight or the second weight is obtained by training based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

13. The apparatus of claim 10, wherein the instructions further cause the processor to be configured to perform superposition, convolution, and pooling processing on the third feature map and the second first feature map to obtain the at least one second feature map.

14. The apparatus of claim 9, wherein the instructions further cause the processor to be configured to: perform a second convolution processing on the at least two second feature maps to obtain features of the at least two second feature maps; and splice the features to obtain the saliency map of the image.

15. The apparatus of claim 9, wherein the instructions further cause the processor to be configured to perform a first guided filtering on the saliency map of the image based on the image to obtain a segmented image of the image.

16. The apparatus of claim 15, wherein the saliency map is a first saliency map, wherein a resolution of the first saliency map is lower than the resolution of the image, and wherein the instructions further cause the processor to be configured to: upsample the first saliency map to obtain a second saliency map, wherein a resolution of the second saliency map is the same as the resolution of the image; and perform a second guided filtering on the second saliency map based on the image to obtain the segmented image.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a schematic diagram of a network architecture of an existing method for detecting a salient object in an image;

(2) FIG. 2 is a schematic flowchart of a method for detecting a salient object in an image according to an embodiment of this application;

(3) FIG. 3 is a schematic diagram of a convolutional neural network architecture according to an embodiment of this application;

(4) FIG. 4 is a schematic diagram of processing a first feature map;

(5) FIG. 5 is a diagram of comparison between a saliency map obtained in an embodiment of this application and a saliency map obtained in another method;

(6) FIG. 6 is a schematic diagram of a method for detecting a salient object in an image according to an embodiment of this application;

(7) FIG. 7 is a diagram of comparison between a saliency map obtained in an embodiment of this application and a saliency map obtained in another method;

(8) FIG. 8 is a schematic block diagram of an apparatus for detecting a salient object in an image according to an embodiment of this application; and

(9) FIG. 9 is a schematic block diagram of an apparatus for detecting a salient object in an image according to an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

(10) The following describes the technical solutions in this application with reference to the accompanying drawings.

(11) FIG. 2 is a schematic flowchart of a method for detecting a salient object in an image according to an embodiment of this application. The method in FIG. 2 may be performed in a network architecture of a convolutional neural network. The method in FIG. 2 includes the following steps.

(12) Step 210. Separately perform convolution processing corresponding to at least two convolutional layers on a to-be-processed image to obtain at least two first feature maps of the to-be-processed image, where resolutions of the at least two first feature maps are lower than a resolution of the to-be-processed image, and resolutions of any two of the at least two first feature maps are different.

(13) A size of a convolution kernel used when convolution processing is performed on the to-be-processed image may be 1. In addition, a function of performing convolution processing may be extracting a feature map required for saliency segmentation from the to-be-processed image. Then, the extracted feature map is further processed to obtain a saliency map of the to-be-processed image.

(14) The to-be-processed image may be an original image that needs to be processed, or may be an image obtained after downsampling processing is performed on the original image. The original image is first downsampled before the saliency map of the image is obtained, thereby reducing a resolution of the image, and decreasing complexity of subsequent image processing.

(15) The resolution of the first feature map may be lower than the resolution of the to-be-processed image. For example, if the to-be-processed image is an image whose resolution is 256×256, the resolution of the first feature map may be 128×128, 64×64, 32×32, 16×16, 8×8, or the like.

(16) In addition, when convolution processing is performed on the to-be-processed image, convolution processing may be separately performed on the to-be-processed image at different convolutional layers to obtain first feature maps with different resolutions. For example, the to-be-processed image is an image whose resolution is 256×256, and after convolution processing is performed on the to-be-processed image at four convolutional layers, four first feature maps whose resolutions are respectively 64×64, 32×32, 16×16, and 8×8 are obtained.

(17) Step 220. Perform superposition processing on at least two first feature maps included in a superposition set in at least two sets to obtain at least two second feature maps of the to-be-processed image, where the at least two sets are respectively corresponding to different resolutions, the at least two sets are in a one-to-one correspondence with the at least two second feature maps, and a resolution of a first feature map included in the superposition set is lower than or equal to a resolution of a second feature map corresponding to the superposition set.

(18) It should be understood that the superposition set in step 220 may be a set in the at least two sets that includes at least two first feature maps. In addition, the at least two sets may include another set in addition to the superposition set. For example, the at least two sets may also include a set including only one first feature map. When a specific set includes only one first feature map, the first feature map included in the set is not superposed, but the first feature map may be directly determined as a second feature map corresponding to the set. In addition, it should be further understood that a resolution corresponding to each of the at least two sets may be a resolution of a second feature map obtained after superposition processing is performed on first feature maps in the set.

(19) For example, when superposition processing is performed on at least two first feature maps included in a specific superposition set, a most salient region of a first feature map with a higher resolution may be located based on a first feature map with a lower resolution in the superposition set. In addition, sparsity and irregularity of the first feature map with the lower resolution may also be improved based on the first feature map with the higher resolution. In this way, a second feature map finally obtained by performing superposition processing on the at least two first feature maps in the superposition set can better display a salient region in an image. Then, after at least two second feature maps obtained after performing superposition processing on the at least two sets are spliced, a saliency map with a better effect can be obtained.

(20) The following describes obtaining of the at least two second feature maps in the at least two sets with reference to a specific case. For example, four first feature maps are obtained after convolution processing is performed on the to-be-processed image, and the four first feature maps are respectively A, B, C, and D. Resolutions of A, B, C, and D are respectively 64×64, 32×32, 16×16, and 8×8. Resolutions corresponding to a set 1 to a set 4 are respectively 64×64, 32×32, 16×16, and 8×8. In this case, the set 1 includes A, B, C, and D, the set 2 includes B, C, and D, the set 3 includes C and D, and the set 4 includes only D. The set 1 to the set 3 each include at least two first feature maps. Therefore, the set 1 to the set 3 may be referred to as superposition sets. Because the set 4 includes only one first feature map, the set 4 is not a superposition set. For the set 1, a second feature map corresponding to the set 1 may be obtained by superposing A, B, C, and D. For the set 2, a second feature map corresponding to the set 2 may be obtained by superposing B, C, and D. For the set 3, a second feature map corresponding to the set 2 may be obtained by superposing C and D. For the set 4, because only D is included, D may be directly determined as a second feature map corresponding to the set 4.

(21) Optionally, separately performing superposition processing on at least two first feature maps included in a superposition set in at least two sets in step 220 includes upsampling a first feature map, in the superposition set, whose resolution is lower than the resolution of the second feature map corresponding to the superposition set to obtain at least two third feature maps whose resolutions are the same as the resolution of the second feature map corresponding to the superposition set, where the at least two third feature maps are in a one-to-one correspondence with the at least two first feature maps, and superposing the at least two third feature maps to obtain the second feature map corresponding to the superposition set.

(22) It should be understood that resolutions of some first feature maps in the superposition set may be lower than the resolution of the second feature map corresponding to the superposition set. In this case, some first feature maps with lower resolutions are upsampled such that resolutions of all first feature maps in the superposition set can be the same, and an effect of superposition processing can be ensured.

(23) Optionally, the superposing the at least two third feature maps to obtain the second feature map corresponding to the superposition set includes superposing the at least two third feature maps based on a weight corresponding to each of the at least two third feature maps to obtain the second feature map.

(24) It should be understood that the superposing the at least two third feature maps based on a weight corresponding to each of the at least two third feature maps may be multiplying the weight corresponding to each of the at least two third feature maps by a pixel value of each third feature map, summing up results obtained after multiplying, and using a result obtained by summing up as a pixel value of the second feature map to obtain the second feature map.

(25) For example, after three first feature maps included in a specific superposition set are processed, three third feature maps are obtained. It is assumed that the three feature maps are respectively X, Y, and Z, and weights of X, Y, and Z are respectively 30%, 30%, and 40%. When X, Y, and Z are superposed, 30% of a pixel value of X, 30% of a pixel value of Y, and 40% of a pixel value of Z are summed up, and a result obtained by summing up is used as a pixel value of a second feature map W obtained after the superposition.

(26) In addition, the weight of each of at least one third feature map may be obtained by training based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(27) A procedure of obtaining the saliency map of the training image may be consistent with the procedure of obtaining the saliency map of the to-be-processed image. Therefore, before the to-be-processed image is processed, the saliency map of the training image may be first obtained according to the procedure of the method in the first aspect, and then the weight of each of the at least one third feature map is trained based on the difference between the saliency map of the training image and the reference saliency map corresponding to the training image to obtain the weight of each third feature map.

(28) Step 230. Splice the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(29) In this application, after the at least two first feature maps are obtained through the convolution processing, different from that in the other approaches in which a final saliency map is obtained by directly performing superposition processing on the at least two first feature maps, the at least two sets are first determined based on the resolutions, the feature maps included in the superposition set in the at least two sets are superposed, and then the second feature maps obtained in the sets are spliced to obtain the saliency map of the to-be-processed image. In addition, in processes of superposing and splicing, features of the feature maps with different resolutions are fully considered, thereby obtaining a saliency map with a better effect.

(30) For example, when superposition processing is performed on at least two first feature maps included in a specific superposition set, a most salient region of a first feature map with a higher resolution may be located based on a first feature map with a lower resolution in the superposition set. In addition, sparsity and irregularity of the first feature map with the lower resolution may also be improved based on the first feature map with the higher resolution. In this way, a second feature map finally obtained by performing superposition processing on the at least two first feature maps in the superposition set can better display a salient region in an image. Then, after at least two second feature maps obtained after performing superposition processing on the at least two sets are spliced, a saliency map with a better effect can be obtained.

(31) Optionally, in an embodiment, splicing the at least two second feature maps to obtain the saliency map of the to-be-processed image in step 230 includes splicing the at least two second feature maps based on a weight corresponding to each of the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(32) It should be understood that the splicing the at least two second feature maps based on a weight corresponding to each of the at least two second feature maps may be multiplying the weight corresponding to each of the at least two second feature maps by a pixel value of each second feature map, summing up results of multiplying the weight by the pixel value, and using a pixel value result obtained by summing up as a pixel value of the saliency map of the to-be-processed image to obtain the saliency map of the to-be-processed image.

(33) It should be understood that the weight of each of the at least two second feature maps may be determined based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(34) In addition, a procedure of obtaining the saliency map of the training image may be consistent with the procedure of obtaining the saliency map of the to-be-processed image. Therefore, before the to-be-processed image is processed, the saliency map of the training image may be first obtained according to the procedure of the method in the first aspect, and then the weight of each of the at least two second feature maps is trained based on the difference between the saliency map of the training image and the reference saliency map corresponding to the training image to obtain the weight of each second feature map.

(35) It should be understood that, the reference saliency map corresponding to the training image may be a human-annotated saliency map, or a saliency map with a better effect that is recognized by a machine. The difference between the saliency map of the training image and the reference saliency map corresponding to the training image may be represented using a function value of a loss function. In a weight training process, the function value of the loss function may be reversely propagated in a convolutional neural network, and all weights are adjusted. The weights may be adjusted along a direction in which function values of the loss function are in descending order, until a global optimal solution is achieved (a final adjustment result may be that the function value of the loss function is minimum or the function value of the loss function is less than a specific threshold).

(36) In the foregoing method in FIG. 2, there are four sets in total, and the four sets may correspond to a path 1 to a path 4 shown in FIG. 3. An output loss function used when a first feature map corresponding to each path is processed on the path is l.sub.side.sup.m, m=1, . . . . ,4. In this case, an output loss function of all paths is L.sub.side=Σ.sub.m=1.sup.4α.sub.ml.sub.side.sup.(m), where α.sub.m is an output loss weight on an m.sup.th path, and a loss function used when a path fusion module processes four second feature maps output on the four paths is L.sub.fuse. In this case, a final loss function for processing the to-be-processed image is L.sub.final=L.sub.fuse+L.sub.side. The final loss function herein may indicate the difference between the saliency map of the training image and the reference saliency map corresponding to the training image.

(37) Optionally, in an embodiment, performing superposition processing on at least two first feature maps included in a superposition set in at least two sets in step 220 includes performing superposition processing on the at least two first feature maps included in the superposition set in the at least two sets, performing convolution processing on at least two feature maps obtained after the superposition processing, to obtain at least two feature maps on which convolution processing is performed, where the convolution processing is used to extract features of the at least two feature maps obtained after the superposition processing, and performing pooling processing on the at least two feature maps obtained through the convolution processing to obtain the at least two second feature maps in step 220.

(38) It should be understood that a size of a convolution kernel used when convolution processing is performed on the at least two feature maps obtained after the superposition processing may be 1. The at least two feature maps obtained after the superposition processing can be collected in an integrated manner by performing convolution processing on the at least two feature maps, to highlight high-value features of the feature maps.

(39) A feature of an image obtained after the superposition can be further extracted by performing convolution processing and pooling processing on the at least two feature maps obtained after the superposition, and the extracted feature is used as the second feature map such that a calculation amount of subsequent processing can be reduced. In addition, a feature with a relatively low value can be excluded through feature extraction such that an effect of the finally obtained saliency map is improved.

(40) Optionally, in an embodiment, splicing the at least two second feature maps of the to-be-processed image in step 230 includes performing convolution processing on the at least two second feature maps to obtain features of the at least two second feature maps, and splicing the features of the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(41) A size of a convolution kernel used when convolution processing is performed on the at least two second feature maps may be 1. The features of the second feature maps can be further extracted through the convolution processing. In this way, local features in a processed image can be more distinguishable from each other, thereby achieving a better saliency detection effect.

(42) Before maps are spliced, convolution processing is first performed on the maps. A map feature can be further extracted, and an extracted feature map is used as a basis of subsequent splicing, thereby reducing complexity of the subsequent splicing. In addition, a feature with a relatively low value can be excluded through feature extraction such that an effect of the finally obtained saliency map is improved.

(43) It should be understood that, in step 220 and step 230, superposition processing performed on the at least two first feature maps included in the superposition set in the at least two sets is equivalent to superposition processing separately performed on at least two first feature maps along different paths, and splicing the at least two second feature maps is equivalent to splicing second feature maps obtained from at least two different paths. For example, as shown in FIG. 3, a convolutional neural network architecture includes four layers (each layer is equivalent to a convolutional layer), four paths, and a path fusion module. Convolution processing is separately performed on a to-be-processed image (a resolution of the to-be-processed image shown in FIG. 3 is 256×256) at a layer 1 to a layer 4, and four first feature maps whose resolutions are respectively 64×64, 32×32, 16×16, and 8×8 are obtained. Then, a first feature map on each path is processed from the path 1 to the path 4, and a specific process includes performing, along the path 1, superposition processing on the four first feature maps whose resolutions are respectively 64×64, 32×32, 16×16, and 8×8 to obtain a second feature map, performing, along the path 2, superposition processing on three first feature maps whose resolutions are respectively 32×32, 16×16, and 8×8 to obtain a second feature map, performing, along the path 3, superposition processing on two first feature maps whose resolutions are respectively 16×16 and 8×8 to obtain a second feature map, and processing, along the path 4, a first feature map whose resolution is 8×8 to obtain a second feature map (on the path 4, the first feature map whose resolution is 8×8 may be directly determined as the second feature map corresponding to the path 4). Finally, four second feature maps are obtained on the four paths. Then, the path fusion module splices the second feature maps on the path 1 to the path 4 to obtain a saliency map of the to-be-processed image. In addition, it should be further understood that, when the first feature map is processed on each path in the path 1 to the path 4, the first feature map on each path has a corresponding weight, and when the path fusion module splices the second feature maps on the path 1 to the path 4, the second feature map obtained from each path also has a respective weight. The weights may be trained based on a function value of a loss function, to obtain new weights. The function value of the loss function may be reversely propagated in the architecture in FIG. 3, and the weights may be adjusted along a direction in which function values of the loss function are in descending order, until a global optimal solution is achieved (a final adjustment result may be that the function value of the loss function is minimum or the function value of the loss function is less than a specific threshold).

(44) In addition, in the architecture shown in FIG. 3, after superposition processing is performed on at least two first feature maps on a specific path, a second feature map corresponding to the path is not directly obtained. Convolution processing and pooling processing are further performed on the feature map obtained after the superposition processing on the path, and then the second feature map corresponding to the path is obtained. Similarly, before splicing the second feature maps on the path 1 to the path 4, the path fusion module may further first perform convolution processing on the second feature maps on the path 1 to the path 4, and then splice the second feature maps on the path 1 to the path 4.

(45) It should be understood that, if a specific superposition set includes four first feature maps whose resolutions are respectively 64×64, 32×32, 16×16, and 8×8, processing the four first feature maps in the superposition set is equivalent to processing the four first feature maps whose resolutions are respectively 64×64, 32×32, 16×16, and 8×8 on the path 1 in FIG. 3. With reference to FIG. 4, the following describes in detail a process of processing the four first feature maps on the path 1.

(46) As shown in FIG. 4, first feature maps at a layer 1 to a layer 4 are respectively obtained (resolutions of the first feature maps obtained at the layer 1 to the layer 4 are respectively 64×64, 32×32, 16×16, and 8×8). Upsampling processing is separately performed, based on the resolution 64×64, on first feature maps obtained at the layer 2 to the layer 4 (because a resolution of a first feature map obtained at the layer 1 is 64×64, the first feature map obtained at the layer 1 is no longer upsampled, but normal sampling may be directly performed on the first feature map obtained at the layer 1 to obtain a third feature map), and four third feature maps whose resolutions are all 64×64 are finally obtained. Then, the four third feature maps whose resolutions are all 64×64 are superposed, to obtain a fourth feature map. Finally, convolution processing and pooling processing are performed on the fourth feature map. In addition, an activation function such as a current linear rectification function or a rectified linear unit (ReLU) may be used to finely adjust an image obtained through the convolution processing and the pooling processing to finally obtain a second feature map corresponding to the path 1.

(47) Optionally, performing convolution and splicing on the at least two second feature maps of the to-be-processed image to obtain the saliency map of the to-be-processed image includes upsampling the at least two second feature maps to obtain at least two fifth feature maps whose resolutions are the same, and performing convolution and splicing on the at least two fifth feature maps to obtain the saliency map of the to-be-processed image.

(48) Table 1 shows a result of comparison between detection data in the method for detecting a salient object in an image in this embodiment of this application and detection data in other methods. “Ours” indicates the detection data in the method of this application. RC[7], CHM[29], DSR[30], DRFI[22], MC[49], ELD[12], MDF[27], DS[12], RFCN[45], DHS[34], and DCL[28] correspond to the detection data in the other methods. In addition, a larger F.sub.β value and a smaller mean squared error (MSE) value indicate better algorithm performance. It can be learned from the data in Table 1 that F.sub.β values in the method in this embodiment of this application are basically larger than F.sub.β values in the other methods, and MAE values in the method in this embodiment of this application are basically less than MAE values in the other methods. Therefore, a better effect is achieved in the method in this embodiment of this application.

(49) TABLE-US-00001 TABLE 1 MSRA-B[35] ECCSD[48] HKU-IS[27] ECCSD[32] SOD[37, 38] Data Method F.sub.β MAE F.sub.β MAE F.sub.β MAE F.sub.β MAE F.sub.β MAE  RC[7] 0.817 0.138 0.741 0.187 0.726 0.165 0.640 0.225 0.657 0.242  CHM[29] 0.809 0.138 0.722 0.195 0.728 0.158 0.631 0.222 0.655 0.249   DSR[30] 0.812 0.119 0.737 0.173 0.735 0.140 0.646 0.204 0.655 0.234  DRFI[22] 0.855 0.119 0.787 0.166 0.783 0.143 0.679 0.221 0.712 0.215   MC[49] 0.872 0.062 0.822 0.107 0.781 0.098 0.721 0.147 0.708 0.184   ELD[12] 0.914 0.042 0.865 0.981 0.844 0.071 0.767 0.121 0.760 0.154  MDF[27] 0.885 0.104 0.833 0.108 0.860 0.129 0.764 0.145 0.785 0.155   DS[12] — — 0.810 0.160 — — 0.818 0.170 0.781 0.150 RFCN[45] 0.926 0.062 0.898 0.097 0.895 0.079 0.827 0.118 0.805 0.161  DHS[34] — — 0.905 0.061 0.892 0.052 0.820 0.091 0.823 0.127  DCL[28] 0.916 0.047 0.898 0.071 0.907 0.048 0.822 0.108 0.823 0.126 Ours 0.926 0.032 0.913 0.057 0.913 0.040 0.831 0.083 0.842 0.121

(50) In addition, FIG. 5 shows a result of comparison between the saliency map obtained by processing the original image according to the method for detecting a salient object in an image in this embodiment of this application and saliency maps obtained by processing an original image according to other methods. DCL, DHS, RFCN, DS, MDF, ELD, MC, DRFI, and DSR correspond to the saliency maps obtained by processing the original image according to the other methods. It can be learned from FIG. 5 that, compared with the other methods, the saliency map obtained in the method in this application is closer to a real saliency map (the real saliency map may be obtained through human annotation). Therefore, the saliency map obtained in the method in this application has a better effect.

(51) After the saliency map of the to-be-processed image is obtained, a segmented image of the to-be-processed image may be further obtained with reference to the to-be-processed image and the saliency map of the to-be-processed image. Guided filtering may be performed on the saliency map of the to-be-processed image based on the to-be-processed image to obtain the segmented image of the to-be-processed image. It should be understood that the segmented image of the to-be-processed image may also be considered as a type of saliency distribution map.

(52) An image segmentation edge in the saliency map of the to-be-processed image can be further optimized through guided filtering, thereby obtaining a segmented image with a better effect.

(53) In addition, it is assumed that the saliency map of the to-be-processed image is a first saliency map, and a resolution of the first saliency map is lower than the resolution of the to-be-processed image. In this case, the first saliency map may be first upsampled to obtain a second saliency map whose resolution is the same as that of the to-be-processed image. Then, guided filtering is performed on the second saliency map based on the to-be-processed image, to obtain the segmented image of the to-be-processed image.

(54) For example, if a resolution of the saliency map of the to-be-processed image is 64×64 and the resolution of the to-be-processed image is 256×256, the resolution of the saliency map of the to-be-processed image may be first adjusted to 256×256, and then guided filtering is performed on the saliency map of the to-be-processed image based on the to-be-processed image, to obtain the segmented image of the to-be-processed image.

(55) FIG. 6 is a schematic diagram of a method for detecting a salient object in an image according to an embodiment of this application. The method in FIG. 6 includes the following steps.

(56) Step 310. Obtain an original image.

(57) The original image herein may be a photo that includes a portrait, and the photo may be a photo obtained by taking a selfie using a mobile phone.

(58) Step 320. Downsample the original image, and then output a saliency map with a low resolution using a trained convolutional neural network model.

(59) For example, if a resolution of the original image is 256×256, the original image may be first upsampled to obtain an image whose resolution is 128×128.

(60) It should be understood that the convolutional neural network model herein may be shown in FIG. 3. When the convolutional neural network model is trained, different datasets may be used according to different scenarios. For example, during portrait segmentation, a convolutional neural network model may be trained using a portrait segmentation dataset, and during vehicle segmentation, the convolutional neural network model may be trained using a vehicle segmentation dataset.

(61) Step 330. Upsample the saliency map obtained in step 320 to obtain a saliency map whose size is the same as that of the original image.

(62) For example, if a resolution of the original image is 256×256 and a resolution of the saliency map obtained in step 320 is 128×128, the resolution of the saliency map may be adjusted from 128×128 to 256×256 by upsampling.

(63) Step 340. Guided filtering is performed, based on the original image, on the saliency map finally obtained in step 330 to obtain a segmented image of the to-be-processed image.

(64) It should be understood that an image edge of the saliency map obtained in step 330 can be optimized using the guided filtering to obtain a segmented image with a better effect.

(65) The segmented image of the original image may be obtained by performing the foregoing steps 310 to 340. Then, processing such as beautifying a portrait and highlighting a circle may be performed on the original image based on the segmented image of the original image to implement beautification processing of the original image and improve a display effect of the original image.

(66) It should be understood that the method for detecting the salient object in the image in this embodiment of this application can implement segmentation of an object in an image in a plurality of scenarios, for example, can implement segmentation of an important target such as a person, a vehicle, or an animal in an image. Application of the method for detecting the salient object in the image in this embodiment of this application in two relatively common scenarios, namely, portrait segmentation and vehicle segmentation, is described in detail below with reference to Example 1 and Example 2.

(67) Example 1: Portrait segmentation includes the following steps.

(68) Step 410. Train a convolutional neural network model using a portrait segmentation dataset.

(69) The convolutional neural network model may be shown in FIG. 3.

(70) The portrait segmentation dataset includes a portrait picture (picture including a portrait) and a real saliency distribution map corresponding to the portrait picture. In addition, to improve a training effect, processing such as mirroring, rotation, and illumination changing may be further performed on the picture to avoid overfitting during convolutional neural network training.

(71) Step 420. First downsample an input portrait picture I.sub.h to obtain a picture I.sub.l with a low resolution, process, using the trained convolutional neural network, the picture with the low resolution obtained by downsampling, and finally output a portrait segmented image S.sub.l with a low resolution.

(72) The input portrait picture is first downsampled, thereby reducing an image resolution, and reducing complexity of subsequent image processing.

(73) Step 430. Upsample the portrait segmented image S.sub.l to obtain a picture S.sub.h whose size is the same as that of the original portrait picture.

(74) Step 440. Perform guided filtering on the picture S.sub.h based on the portrait picture I.sub.h to obtain a final portrait segmented image.

(75) It is assumed that a guided filtering function is ƒ(.), and an output image after filtering is {tilde over (S)}.sub.h=ƒ(I.sub.h, S.sub.h, r, eps), where r is a filtering radius, and eps is a smooth parameter. A portrait segmentation edge is further optimized using the guided filtering such that the edge of the portrait segmented image is clearer.

(76) In the other approaches, during portrait segmentation, a portrait edge cannot be accurately attached, and erroneous detection or missed detection may occur in a local area in an image. However, in the method in this application, a portrait can be accurately located in a complex scenario, and an edge of the portrait can be relatively accurately attached, to achieve a better segmentation effect.

(77) The input picture is first upsampled. An image with a lower resolution is obtained, and basic portrait segmentation is implemented on the image with the lower resolution. In addition, in the method in this application, a portrait in an image can be automatically detected without manual interaction, and the portrait segmentation is implemented.

(78) A result of comparison between the image segmentation in the method in this application and image segmentation in the other approaches is shown in FIG. 7. It can be learned from FIG. 7 that, compared with the existing method, in the method for detecting the salient object in the image in this embodiment of this application, an object region having a salient feature in the image can be accurately distinguished, an effect of saliency analysis is better.

(79) Example 2: Vehicle segmentation includes the following steps.

(80) Step 510. Train a convolutional neural network model using a vehicle segmentation dataset.

(81) Step 520. First downsample an input road scene picture I.sub.h to obtain a picture I.sub.l with a low resolution, process, using the trained convolutional neural network, the picture with the low resolution obtained by downsampling, and finally output a vehicle segmented image S.sub.l with a low resolution.

(82) Step 530. Upsample the vehicle segmented image S.sub.l to obtain a picture S.sub.h whose size is the same as that of the original road scene picture.

(83) Step 540. Perform guided filtering on the picture S.sub.h based on the road scene picture I.sub.h to obtain a final vehicle segmented image.

(84) It should be understood that the foregoing is merely two scenarios applied to the method for detecting the salient object in the image in the embodiments of this application. In essence, the method for detecting the salient object in the image in the embodiments of this application may also be applied to another scenario. Providing that training data in the scenario is used to train the convolutional neural network and corresponding processing is performed on the to-be-processed image, a better effect can also be achieved.

(85) In the method in this application, the vehicle segmentation can be basically implemented on an image with a lower resolution, semantic accuracy can also be ensured in a complex and changeable background environment, and finally a detailing degree of an edge can be ensured by performing guided filtering on an image with a high resolution. Without manual interaction, a vehicle in an image can be automatically detected and segmented, and automatic driving can be assisted to decide. Compared with another existing method, in the present application, a vehicle edge can be effectively segmented, to improve a determining capability of estimating vehicle bit posture, a vehicle distance, and the like.

(86) The following describes in detail a method for detecting a salient object in an image in an embodiment of this application with reference to FIG. 3 again.

(87) The model shown in FIG. 3 is a basic network model based on a ResNet-101 architecture. In the network model, there are four layers in total, four paths (the paths herein are equivalent to the foregoing sets), and one multi-path fusion module. Resolutions corresponding to a layer 1 to a layer 4 are respectively 64×64, 32×32, 16×16, and 8×8. A feature map of at least one level of four levels is received on each of the four paths as an input. Feature maps of four levels (level 1 to level 4) are received on a path 1, feature maps of three levels (level 2 to level 4) are received on a path 2, feature maps of two layers (layer 3 and layer 4) are received on a path 4, and a feature map of one layer (layer 4) is received on a path 4.

(88) It is assumed that a resolution of the to-be-processed image is 256×256. The following describes in detail specific operations of layers, paths, and the path fusion module in the network model shown in FIG. 3.

(89) Step 610. Feature maps of corresponding resolutions are obtained from the to-be-processed image at the layer 1 to the layer 4.

(90) Feature maps whose resolutions are 64×64, 32×32, 16×16, 8×8 are respectively obtained from the to-be-processed image at the layer 1 to the layer 4.

(91) Step 620. Feature maps of at least one level are separately fused on the path 1 to path 4.

(92) Using the path 1 as an example, feature maps of a layer 1 to a layer 4 are received on a path 1, and the feature maps on the layer 1 to the layer 4 are upsampled, to obtain four images whose resolutions are the same. Then, the four images with the same resolution are fused, a fused feature map is obtained, convolution processing and pooling processing are performed on the fused feature map. Finally, a feature map after the convolution processing and the pooling processing is finely adjusted using a linear rectification function, to finally obtain a feature map on the path 1.

(93) Step 630. The multi-path fusion module fuses the feature maps on the path 1 to the path 4.

(94) The multi-path fusion module upsamples the feature maps on the path 1 to the path 4, to obtain four feature maps whose resolutions are all 64×64, performs a convolution operation and a splicing operation on the four feature maps, and upsamples a feature map obtained through the convolution operation and the splicing operation to a size of the to-be-processed image (a resolution is 128×128) to obtain a saliency map of the to-be-processed image.

(95) It should be understood that the network architecture shown in FIG. 3 is merely a possible architecture of the method in the embodiments of this application. Actually, several improvements and replacements may be further made based on the network architecture shown in FIG. 3. For example, a quantity of convolutional layers and a quantity of paths are changed, a correspondence between a path and a convolutional layer is changed, and the like. Network architectures obtained through improvement and replacement shall fall within the protection scope of this application.

(96) The method for detecting the salient object in the image in the embodiments of this application may be applied to segmenting an important target in a picture. For example, in a portrait mode, a portrait and a background object in a picture are segmented from each other, and different processing is performed on the portrait and the background (for example, face beautification processing is performed on the portrait, blurring processing is performed on the background, a color of the background is enhanced, and a dark area is compressed at four corners of the background), thereby achieving an artistic effect of highlighting and beautifying a portrait. In the method in this embodiment of this application, a self-portrait effect and a wide aperture effect in the portrait mode may be applied. The method in this embodiment of this application may be further applied to portrait stylization, portrait beautification, portrait background editing and composition (for example, generation of an identification photo, and composition of a group photo at a scenic spot). After saliency analysis is performed on an original picture, a saliency map of the original picture is obtained. Then, stylization processing may be performed on the portrait in the original picture based on the saliency map obtained through analysis, or beautification processing may be performed on the portrait, or the background in the original picture is replaced.

(97) Optionally, the method in this embodiment of this application may be further applied to segmenting and recognizing an object of interest in the image, and the like.

(98) The foregoing describes in detail the method for detecting the salient object in the image in the embodiments of this application with reference to FIG. 2 to FIG. 7. The following describes an apparatus for detecting a salient object in an image in an embodiment of this application with reference to FIG. 8 and FIG. 9. It should be understood that, the apparatuses in FIG. 8 and FIG. 9 can perform corresponding steps of the foregoing method for detecting the salient object in the image. For brevity, repeated descriptions are appropriately omitted below.

(99) FIG. 8 is a schematic block diagram of an apparatus for detecting a salient object in an image according to an embodiment of this application. An apparatus 800 in FIG. 8 includes a convolution module 810 configured to separately perform convolution processing corresponding to at least two convolutional layers on a to-be-processed image to obtain at least two first feature maps of the to-be-processed image, where resolutions of the at least two first feature maps are lower than a resolution of the to-be-processed image, and resolutions of any two of the at least two first feature maps are different, a superposition module 820 configured to perform superposition processing on at least two first feature maps included in a superposition set in at least two sets to obtain at least two second feature maps of the to-be-processed image, where the at least two sets are respectively corresponding to different resolutions, the at least two sets are in a one-to-one correspondence with the at least two second feature maps, and a resolution of a first feature map included in the superposition set is lower than or equal to a resolution of a second feature map corresponding to the superposition set, and a splicing module 830 configured to splice the at least two second feature maps to obtain a saliency map of the to-be-processed image.

(100) Optionally, in an embodiment, the superposition module 820 is configured to upsample a first feature map, in the superposition set, whose resolution is lower than the resolution of the second feature map corresponding to the superposition set, to obtain at least two third feature maps whose resolutions are the same as the resolution of the second feature map corresponding to the superposition set, where the at least two third feature maps are in a one-to-one correspondence with the at least two first feature maps, and superpose the at least two third feature maps to obtain the second feature map corresponding to the superposition set.

(101) Optionally, in an embodiment, the superposition module 820 is configured to superpose the at least two third feature maps based on a weight corresponding to each of the at least two third feature maps to obtain the second feature map.

(102) Optionally, in an embodiment, the weight of each of at least one third feature map is obtained by training based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(103) Optionally, in an embodiment, the splicing module 830 is configured to splice the at least two second feature maps based on a weight corresponding to each of the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(104) Optionally, in an embodiment, the weight of each of the at least two second feature maps is determined based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(105) Optionally, in an embodiment, the superposition module 820 is configured to perform superposition processing on the at least two first feature maps included in the superposition set in the at least two sets, perform convolution processing on at least two feature maps obtained after the superposition processing to obtain at least two feature maps on which convolution processing is performed, where the convolution processing is used to extract features of the at least two feature maps obtained after the superposition processing, and perform pooling processing on the at least two feature maps obtained through the convolution processing to obtain the at least two second feature maps.

(106) Optionally, in an embodiment, the splicing module 830 is configured to perform convolution processing on the at least two second feature maps to obtain features of the at least two second feature maps, and splice the features of the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(107) Optionally, in an embodiment, the apparatus 800 further includes a filtering module (not shown) configured to perform guided filtering on the saliency map of the to-be-processed image based on the to-be-processed image to obtain a segmented image of the to-be-processed image.

(108) Optionally, in an embodiment, the saliency map is a first saliency map, a resolution of the first saliency map is lower than the resolution of the to-be-processed image, and the filtering module is configured to upsample the first saliency map to obtain a second saliency map whose resolution is the same as the resolution of the to-be-processed image, and perform guided filtering on the second saliency map based on the to-be-processed image to obtain the segmented image of the to-be-processed image.

(109) FIG. 9 is a schematic block diagram of an apparatus for detecting a salient object in an image according to an embodiment of this application. An apparatus 900 in FIG. 9 includes a memory 910 configured to store a program, and a processor 920 configured to execute the program stored in the memory 910. When the program of the memory 910 is executed, the processor 920 is configured to separately perform convolution processing corresponding to at least two convolutional layers on a to-be-processed image to obtain at least two first feature maps of the to-be-processed image, where resolutions of the at least two first feature maps are lower than a resolution of the to-be-processed image, and resolutions of any two of the at least two first feature maps are different, perform superposition processing on at least two first feature maps included in a superposition set in at least two sets to obtain at least two second feature maps of the to-be-processed image, where the at least two sets are respectively corresponding to different resolutions, the at least two sets are in a one-to-one correspondence with the at least two second feature maps, and a resolution of a first feature map included in the superposition set is lower than or equal to a resolution of a second feature map corresponding to the superposition set, and splice the at least two second feature maps to obtain a saliency map of the to-be-processed image.

(110) Optionally, in an embodiment, the processor 920 is configured to upsample a first feature map, in the superposition set, whose resolution is lower than the resolution of the second feature map corresponding to the superposition set to obtain at least two third feature maps whose resolutions are the same as the resolution of the second feature map corresponding to the superposition set, where the at least two third feature maps are in a one-to-one correspondence with the at least two first feature maps, and superpose the at least two third feature maps to obtain the second feature map corresponding to the superposition set.

(111) Optionally, in an embodiment, the processor 920 is configured to superpose the at least two third feature maps based on a weight corresponding to each of the at least two third feature maps, to obtain the second feature map.

(112) Optionally, in an embodiment, the weight of each of at least one third feature map is obtained by training based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(113) Optionally, in an embodiment, the processor 920 is configured to splice the at least two second feature maps based on a weight corresponding to each of the at least two second feature maps, to obtain the saliency map of the to-be-processed image.

(114) Optionally, in an embodiment, the weight of each of the at least two second feature maps is determined based on a difference between a saliency map of a training image and a reference saliency map corresponding to the training image.

(115) Optionally, in an embodiment, the processor 920 is configured to perform superposition processing on the at least two first feature maps included in the superposition set in the at least two sets, perform convolution processing on at least two feature maps obtained after the superposition processing to obtain at least two feature maps on which convolution processing is performed, where the convolution processing is used to extract features of the at least two feature maps obtained after the superposition processing, and perform pooling processing on the at least two feature maps obtained through the convolution processing to obtain the at least two second feature maps.

(116) Optionally, in an embodiment, the processor 920 is configured to perform convolution processing on the at least two second feature maps, to obtain features of the at least two second feature maps, and splice the features of the at least two second feature maps to obtain the saliency map of the to-be-processed image.

(117) Optionally, in an embodiment, the processor 920 is further configured to perform guided filtering on the saliency map of the to-be-processed image based on the to-be-processed image, to obtain a segmented image of the to-be-processed image.

(118) Optionally, in an embodiment, the saliency map is a first saliency map, a resolution of the first saliency map is lower than the resolution of the to-be-processed image, and the processor 920 is configured to upsample the first saliency map to obtain a second saliency map whose resolution is the same as the resolution of the to-be-processed image, and perform guided filtering on the second saliency map based on the to-be-processed image to obtain the segmented image of the to-be-processed image.

(119) A person of ordinary skill in the art may be aware that units and algorithm steps in the examples described with reference to the embodiments disclosed in this specification can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on a particular application and a design constraint of a technical solution. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

(120) It may be clearly understood by the person skilled in the art that, for convenience and brevity of description, for specific working processes of the system, apparatus, and unit described in the foregoing, refer to corresponding processes in the foregoing method embodiments. Details are not described herein again.

(121) In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in another manner. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, at least two units or components may be combined or integrated into another system, or some features may be ignored or may not be performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connection may be implemented through some interfaces. The indirect couplings or communication connection between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

(122) The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on at least two network units. Some or all of the units may be selected based on actual requirements to achieve the objective of the solution in the embodiment.

(123) In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or at least two units are integrated into one unit.

(124) When the functions are implemented in a form of a software function unit and sold or used as an independent product, the functions may be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of this application essentially, or the part contributing to the other approaches, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes various mediums that can store program code, such as a universal serial bus (USB) flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, and an optical disc.

(125) The foregoing descriptions are merely specific implementations of this application, but the protection scope of this application is not limited thereto. Any variation or replacement readily figured out by the person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.