Three-dimensional object detection method and system based on weighted channel features of a point cloud

Abstract

A three-dimensional object detection method includes: extracting a target in a two-dimensional image by a pre-trained deep convolutional neural network to obtain a plurality of target objects; determining a point cloud frustum in a corresponding three-dimensional point cloud space based on each target object; segmenting the point cloud in the frustum based on a point cloud segmentation network to obtain a point cloud of interest; and estimating parameters of a 3D box in the point cloud of interest based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection. According to the present invention, the features of the image can be learned more accurately by the deep convolutional neural network and the parameters of the 3D box in the point cloud of interest are estimated based on the network with the weighted channel features.

Claims

1. A three-dimensional object detection method based on weighted channel features of a point cloud, comprising: extracting a target in a two-dimensional image by a pre-trained deep convolutional neural network to obtain a plurality of target objects; determining a point cloud frustum in a three-dimensional point cloud space corresponding to each target object of the plurality of target objects based on the each target object; segmenting a point cloud in the point cloud frustum based on a point cloud segmentation network to obtain a point cloud of interest; and estimating parameters of a 3D box in the point cloud of interest based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection; wherein, the step of determining the point cloud frustum in the three-dimensional point cloud space corresponding to the each target object based on the each target object specifically comprises: converting a point x.sub.vel on a coordinate system of a Lidar sensor into a point x.sub.cam on a coordinate system of a camera by the following formula:
x.sub.cam=T.sub.vel.sup.camx.sub.vel; wherein, T.sub.vel.sup.cam represents a transformation matrix from the coordinate system of the Lidar sensor to the coordinate system of the camera; projecting the point x.sub.cam on the coordinate system of the camera to a pixel point x.sub.img on the two-dimensional image by the following formula:
x.sub.img=Px.sub.cam; wherein, P represents a projection matrix from the coordinate system of the camera to a coordinate system of the two-dimensional image; and determining a plurality of point clouds projected onto the each target object according to the each target object, wherein the plurality of point clouds projected onto the each target object form the point cloud frustum.

2. The three-dimensional object detection method according to claim 1, wherein, the plurality of target objects are obtained by the following formula: x, y, w, h=Net(I); wherein, I represents the two-dimensional image, and Net represents the pre-trained deep convolutional neural network; and coordinates (x, y) of a center point, a length h and a width w of a 2D box represent a position of the each target object.

3. The three-dimensional object detection method according to claim 1, wherein, the step of segmenting the point cloud in the point cloud frustum based on the point cloud segmentation network to obtain the point cloud of interest specifically comprises: calculating a probability that the point cloud in the point cloud frustum belongs to a point cloud of interest based on the point cloud segmentation network by the following formula:
p.sub.i=f(x.sub.i, θ); wherein, x.sub.i represents an i.sup.th point cloud in the point cloud frustum, θ represents a network training parameter, p.sub.i represents a probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest, and f represents the point cloud segmentation network; and determining and obtaining the point cloud of interest according to the probability that each point cloud in the point cloud frustum belongs to the point cloud of interest and a predetermined probability threshold.

4. The three-dimensional object detection method according to claim 3, wherein, the step of determining and obtaining the point cloud of interest according to the probability that the each point cloud in the point cloud frustum belongs to the point cloud of interest and the predetermined probability threshold, specifically comprises: determining that the point cloud is the point cloud of interest if the probability that the point cloud belongs to the point cloud of interest is greater than 0.5; determining that the point cloud is not the point cloud of interest if the probability that the point cloud belongs to the point cloud of interest is less than or equal to 0.5: $Mas k_{i} = {\begin{matrix} 1, & p_{i} > 0.5 \\ 0, & p_{i} \leq 0.5 \end{matrix};$ wherein, Mask.sub.i represents a mask of the i.sup.th point cloud and takes a value of 1 or 0, and p.sub.i represents the probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest point cloud of interest.

5. A three-dimensional object detection method based on weighted channel features of a point cloud, comprising: extracting a target in a two-dimensional image by a pre-trained deep convolutional neural network to obtain a plurality of target objects; determining a point cloud frustum in a three-dimensional point cloud space corresponding to each target object of the plurality of target objects based on the each target object segmenting a point cloud in the point cloud frustum based on a point cloud segmentation network to obtain a point cloud of interest and estimating parameters of a 3D box in the point cloud of interest based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection; wherein, the step of segmenting the point cloud in the point cloud frustum based on the point cloud segmentation network to obtain the point cloud of interest specifically comprises: calculating a probability that the point cloud in the point cloud frustum belongs to a point cloud of interest based on the point cloud segmentation network by the following formula:
p.sub.i=f(x.sub.i, θ); wherein, x.sub.i represents an i.sup.th point cloud in the point cloud frustum, θ represents a network training parameter, p.sub.i represents a probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest, and f represents the point cloud segmentation network; and determining and obtaining the point cloud of interest according to the probability that each point cloud in the point cloud frustum belongs to the point cloud of interest and a predetermined probability threshold; and wherein, a segmentation loss is generated in the process of segmenting the point cloud in the point cloud frustum; a cross-entropy loss function: $Entropy_Loss (y, y^{*}) = {.Math.}_{i = 1}^{N} y_{i} \log y_{i}^{*} + (1 - y_{i}) \log (1 - y_{i}^{*});$ wherein, y, y* represent a real category and a predicted result, respectively; and N represents a total number of the point cloud in the point cloud frustum; and the segmentation loss:
L.sub.seg=Entropy_Loss(y, y*).

6. The three-dimensional object detection method according to claim 5, wherein, the parameters of the 3D box comprise a center, a size, an angle and a corner; the step of estimating the parameters of the 3D box in the point cloud of interest based on the network with the weighted channel features specifically comprises: calculating a center loss, a size classification loss, a residual size regression loss, an angle classification loss, a residual angle regression loss and a corner loss of the 3D box based on the network with the weighted channel features; calculating a total loss function according to the segmentation loss and the center loss, the size classification loss, the residual size regression loss, the angle classification loss, the residual angle regression loss and the corner loss of the 3D box; and estimating the parameters of the 3D box in the point cloud of interest according to the total loss function to obtain the parameters of the 3D box.

7. The three-dimensional object detection method according to claim 6, wherein, a Hingle loss function is calculated as follows: ${Hingle}_{δ} (x) = {\begin{matrix} 0.5 x^{2}, & .Math. x .Math. \leq δ \\ δ (.Math. x .Math. - 0.5 δ), & .Math. x .Math. > δ \end{matrix};$ (1) the center loss is calculated as follows:
L.sub.center=Hingle.sub.δ=2.0(x); (2) the size classification loss of the 3D box is calculated as follows:
L.sub.size-cls=Entropy_Loss(y, y*); (3) the residual size regression loss of the 3D box is calculated as follows:
L.sub.size-reg=Hingle.sub.δ=1.0(x); (4) the angle classification loss of the 3D box is calculated as follows:
L.sub.angle-cls=Entropy_Loss(y, y*); (5) the residual angle regression loss of the 3D box is calculated as follows: $L_{a n gle - r e g} = {.Math.}_{m}^{S} {.Math.}_{n}^{H} M a s k_{m n} [2 - 2 \cos (θ_{m n} - θ_{m n}^{*})];$ wherein, S represents a number of size templates of the 3D box; m =1, 2, . . . S; H represents a number of direction angle templates of the 3D box; n=1, 2, . . . , H; Mask.sub.mn represents a mask of an m.sup.th size template of the size templates of the 3D box and a mask of an n.sup.th direction angle template of the direction angle templates of the 3D box; θ.sub.mn represents a training parameter of a network in the m.sup.th size template of the 3D box and the n.sup.th direction angle template of the 3D box; and θ*.sub.mn represents a predicted parameter of the network in the m.sup.th size template of the 3D box and the n.sup.th direction angle template of the 3D box; (6) the corner loss of eight corners of the 3D box is calculated as follows: $L_{corner} = {.Math.}_{i = 1}^{S} {.Math.}_{j = 1}^{H} δ_{ij} \min {{.Math.}_{k = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{+} .Math., {.Math.}_{k = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{-} .Math.};$ wherein, S represents the number of the size templates of the 3D box, H represents the number of the direction angle templates of the 3D box, P.sub.k.sup.ij represents a real coordinate of a j.sup.th direction angle template of an i.sup.th size template in a k.sup.th corner, P.sub.k.sup.+ represents a predicted coordinate of the k.sup.th corner, and P.sub.k.sup.− represents a coordinate of the k.sup.th corner after the 3D box is rotated by 180 degrees around a vertical direction; δ.sub.ij represents a real label of the j.sup.th direction template of the i.sup.th size template.

8. The three-dimensional object detection method according to claim 7, wherein, the total loss function L is calculated according to the following formula:
L=L.sub.seg+λ(L.sub.center+L.sub.size-reg+L.sub.size-cls+L.sub.angle-reg+L.sub.angle-cls)+γL.sub.corner; wherein, λ represents a weight of the center loss, the residual size regression loss, the size classification loss, the residual angle regression loss and the angle classification loss of the 3D box, and λ represents a weight of the corner loss.

9. A three-dimensional object detection system based on weighted channel features of a point cloud, comprising: an extraction unit, a determination unit, a segmentation unit, and an estimation unit; wherein the extraction unit is configured to extract a target in a two-dimensional image by a pre-trained deep convolutional neural network to obtain a plurality of target objects; the determination unit is configured to determine a point cloud frustum in a three-dimensional point cloud space corresponding to each target object of the plurality of target objects based on the each target object; the segmentation unit is configured to segment a point cloud in the point cloud frustum based on a point cloud segmentation network to obtain a point cloud of interest; and the estimation unit is configured to estimate parameters of a 3D box in the point cloud of interest based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection; wherein, the step of determining the point cloud frustum in the three-dimensional point cloud space corresponding to the each target object based on the each target object specifically comprises: converting a point x.sub.vel on a coordinate system of a Lidar sensor into a point x.sub.cam on a coordinate system of a camera by the following formula:
x.sub.cam=T.sub.vel.sup.camx.sub.vel; wherein, T.sub.vel.sup.cam represents a transformation matrix from the coordinate system of the Lidar sensor to the coordinate system of the camera; projecting the point x.sub.cam on the coordinate system of the camera to a pixel point x.sub.img on the two-dimensional image by the following formula:
x.sub.img=Px.sub.cam; wherein, P represents a projection matrix from the coordinate system of the camera to a coordinate system of the two-dimensional image; and determining a plurality of point clouds projected onto the each target object according to the each target object, wherein the plurality of point clouds projected onto the each target object form the point cloud frustum.

10. The three-dimensional object detection system according to claim 9, wherein, the plurality of target objects are obtained by the following formula:
x, y, w, h=Net(I); wherein, I represents the two-dimensional image, and Net represents the pre-trained deep convolutional neural network; and coordinates (x, y) of a center point, a length h and a width w of a 2D box represent a position of the each target object.

11. The three-dimensional object detection system according to claim 9, wherein, the step of segmenting the point cloud in the point cloud frustum based on the point cloud segmentation network to obtain the point cloud of interest specifically comprises: calculating a probability that the point cloud in the point cloud frustum belongs to a point cloud of interest based on the point cloud segmentation network by the following formula:
p.sub.i=f(x.sub.i, θ); wherein, x.sub.i represents an i.sup.th point cloud in the point cloud frustum, θ represents a network training parameter, p.sub.i represents a probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest, and f represents the point cloud segmentation network; and determining and obtaining the point cloud of interest according to the probability that each point cloud in the point cloud frustum belongs to the point cloud of interest and a predetermined probability threshold.

12. The three-dimensional object detection system according to claim 11, wherein, the step of determining and obtaining the point cloud of interest according to the probability that the each point cloud in the point cloud frustum belongs to the point cloud of interest and the predetermined probability threshold, specifically comprises: determining that the point cloud is the point cloud of interest if the probability that the point cloud belongs to the point cloud of interest is greater than 0.5; determining that the point cloud is not the point cloud of interest if the probability that the point cloud belongs to the point cloud of interest is less than or equal to 0.5: ${Mask}_{i} = {\begin{matrix} 1, & p_{i} > 0.5 \\ 0, & p_{i} \leq 0.5 \end{matrix};$ wherein, Mask.sub.i represents a mask of the i.sup.th point cloud and takes a value of 1 or 0, and p.sub.i represents the probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest point cloud of interest.

13. The three-dimensional object detection system according to claim 11, wherein, a segmentation loss is generated in the process of segmenting the point cloud in the point cloud frustum; a cross-entropy loss function: $Entropy_Loss (y, y^{⋆}) = {.Math.}_{i = 1}^{N} y_{i} \log y_{i}^{⋆} + (1 - y_{i}) \log (1 - y_{i}^{⋆});$ wherein, y, y* represent a real category and a predicted result, respectively; and N represents a total number of the point cloud in the point cloud frustum; and the segmentation loss:
L.sub.seg=Entropy_Loss(y, y*)

14. The three-dimensional object detection system according to claim 13, wherein, the parameters of the 3 D box comprise a center, a size, an angle and a corner; the step of estimating the parameters of the 3 D box in the point cloud of interest based on the network with the weighted channel features specifically comprises: calculating a center loss, a size classification loss, a residual size regression loss, an angle classification loss, a residual angle regression loss and a corner loss of the 3 D box based on the network with the weighted channel features; calculating a total loss function according to the segmentation loss and the center loss, the size classification loss, the residual size regression loss, the angle classification loss, the residual angle regression loss and the corner loss of the 3 D box; and estimating the parameters of the 3 D box in the point cloud of interest according to the total loss function to obtain the parameters of the 3 D box.

15. The three-dimensional object detection system according to claim 14, wherein, a Hingle loss function is calculated as follows: ${Hingle}_{δ} (x) = {\begin{matrix} 0.5 x^{2}, & .Math. x .Math. \leq δ \\ δ (.Math. x .Math. - 0.5 δ), & .Math. x .Math. > δ \end{matrix};$ (1) the center loss is calculated as follows:
L.sub.center=Hingle.sub.δ=2.0(x); (2) the size classification loss of the 3 D box is calculated as follows:
L.sub.size-reg=Entropy_Loss(y, y*); (3) the residual size regression loss of the 3 D box is calculated as follows:
L.sub.size-reg=Hingle.sub.δ=1.0(x); (4) the angle classification loss of the 3 D box is calculated as follows:
L.sub.angle-cls=Entropy_Loss(y, y*); (5) the residual angle regression loss of the 3 D box is calculated as follows: $L_{angle - reg} = \underset{m}{\overset{S}{.Math.}} \underset{n}{\overset{H}{.Math.}} {Mask}_{mn} [2 - 2 \cos (θ_{mn} - θ_{mn}^{⋆})];$ wherein, S represents a number of size templates of the 3 D box; m=1, 2, . . . S; H represents a number of direction angle templates of the 3 D box; n=1, 2, . . . , H; Mask.sub.mn represents a mask of an m.sup.th size template of the size templates of the 3 D box and a mask of an n.sup.th direction angle template of the direction angle templates of the 3 D box; θ.sub.mn represents a training parameter of a network in the m.sup.th size template of the 3 D box and the n.sup.th direction angle template of the 3 D box; and θ*.sub.mn , represents a predicted parameter of the network in the m.sup.th size template of the 3 D box and the n.sup.th direction angle template of the 3 D box; (6) the corner loss of eight corners of the 3 D box is calculated as follows: $L_{corner} = {.Math.}_{i = 1}^{S} {.Math.}_{j = 1}^{H} δ_{ij} \min {{.Math.}_{m = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{+} .Math., {.Math.}_{m = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{-} .Math.};$ wherein, S represents the number of the size templates of the 3 D box, H represents the number of the direction angle templates of the 3 D box, P.sub.k.sup.ij represents a real coordinate of a j.sup.th direction angle template of an i.sup.th size template in a k.sup.th corner, P.sub.k.sup.ij represents a predicted coordinate of the k.sup.th corner, and P.sub.k.sup.− represents a coordinate of the k.sup.th corner after the 3 D box is rotated by 180 degrees around a vertical direction; δ.sub.ij represents a real label of the j.sup.th direction template of the ith size template.

16. The three-dimensional object detection system according to claim 15, wherein, the total loss function L is calculated according to the following formula:
L=L.sub.seg+λ(L.sub.center+L.sub.size-reg +L.sub.size-clsL.sub.angle-reg+L.sub.angle-cls)+γL.sub.corner; wherein, λ represents a weight of the center loss, the residual size regression loss, the size classification loss, the residual angle regression loss and the angle classification loss of the 3 D box, and y represents a weight of the corner loss.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a flow chart of the three-dimensional object detection method based on the weighted channel features of the point cloud according to the present invention; and

(2) FIG. 2 is a structural schematic diagram of the modules of the three-dimensional object detection system based on the weighted channel features of the point cloud according to the present invention.

REFERENCE NUMERALS

(3) extraction unit-1, determination unit-2, segmentation unit-, and estimation unit-4.

DETAILED DESCRIPTION OF THE EMBODIMENTS

(4) The preferred embodiments of the present invention are described hereinafter with reference to the drawings. It should be understood by those skilled in the art that these embodiments are only intended to illustrate the technical principles of the present invention rather than to limit the scope of protection of the present invention.

(5) The objective of the present invention is to provide a three-dimensional object detection method based on weighted channel features of a point cloud. A target in a two-dimensional image is extracted by a pre-trained deep convolutional neural network so that features of the image can be learned more accurately. The point cloud in a frustum is segmented based on a point cloud segmentation network and parameters of the 3D box in the point cloud of interest are estimated based on a network with the weighted channel features to reduce the weight of features of unimportant points and increase the weight of key points. This has a result of suppressing the interference points and enhancing the key points, thereby improving the precision of the parameters of the 3D box.

(6) In order to make the above-mentioned objectives, features and advantages of the present invention clearer, the present invention will be further described in detail hereinafter with reference to the drawings and specific embodiments.

(7) As shown in FIG. 1, a three-dimensional object detection method based on weighted channel features of a point cloud according to the present invention includes:

(8) Step 100: A target in a two-dimensional image is extracted by a pre-trained deep convolutional neural network to obtain a plurality of target objects.

(9) Step 200: A point cloud frustum in a corresponding three-dimensional point cloud space is determined based on each target object.

(10) Step 300: The point cloud in the frustum is segmented based on a point cloud segmentation network to obtain a point cloud of interest.

(11) Step 400: Parameters of a 3D box in the point cloud of interest are estimated based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection.

(12) In step 100, the plurality of target objects are calculated and obtained according to the following formula:
x, y, w, h=Net(I);

(13) wherein, I represents the two-dimensional image, and Net represents the deep convolutional neural network. The position of each target object is represented by coordinates (x, y) of a center point, a length h and a width w of a 2D box.

(14) In step 200, the step of determining the point cloud frustum in the corresponding three-dimensional point cloud space based on each target object specifically includes:

(15) Step 201: A point x.sub.vel on a coordinate system of a Lidar sensor is converted into a point x.sub.cam on a coordinate system of a camera:
x.sub.cam=T.sub.vel.sup.camx.sub.vel;

(16) wherein, T.sub.vel.sup.camrepresents a transformation matrix of the coordinate system.

(17) Step 202: The point x.sub.cam on the coordinate system of the camera is projected to a pixel point x.sub.img on the two-dimensional image:
x.sub.img=Px.sub.cam;

(18) wherein, P represents a projection matrix from the coordinate system of the camera to a coordinate system of the two-dimensional image.

(19) Step 203: All point clouds projected onto the target object are determined according to each target object, wherein all point clouds projected onto the target object form the point cloud frustum.

(20) For example, the relationship between the point cloud in the Lidar sensor and the target object in the two-dimensional image is established by steps 201-203, so that all point clouds projected onto the target object can be determined, namely, the desired point cloud frustum is obtained.

(21) In step 300, the step of segmenting the point cloud in the frustum based on the point cloud segmentation network to obtain the point cloud of interest specifically includes:

(22) Step 301: The probability that the point cloud in the frustum belongs to the point cloud of interest is calculated based on the point cloud segmentation network:
p.sub.i=f(x.sub.i, θ)

(23) wherein, x.sub.i represents the i.sup.th point cloud in the frustum, θ represents a network training parameter, p.sub.i represents the probability that the i.sup.th point cloud x.sub.i belongs to the point cloud of interest, and f represents the point cloud segmentation network; and

(24) The segmentation network adopts a convolution-deconvolution (Conv-Deconv) symmetric network structure to accurately learn the features of the image.

(25) Step 302: The point cloud of interest is determined and obtained according to the probability that each point cloud belongs to the point cloud of interest and a set probability threshold.

(26) Specifically, if the probability that the point cloud belongs to the point cloud of interest is greater than 0.5, it is determined that the point cloud is the point cloud of interest, otherwise the point cloud is not the point cloud of interest:

(27) ${Mask}_{i} = {\begin{matrix} 1, & p_{i} > 0.5 \\ 0, & p_{i} \leq 0.5 \end{matrix};$

(28) wherein, Mask.sub.i represents a mask of the i.sup.th point cloud and takes a value of 1 or 0.

(29) In addition, a segmentation loss is generated in the process of segmenting the point cloud in the frustum:

(30) The cross-entropy loss function:

(31) $Entropy_Loss (y, y^{*}) = {.Math.}_{i = 1}^{N} y_{i} \log y_{i}^{*} + (1 - y_{i}) \log (1 - y_{i}^{*});$

(32) wherein, y, y* represent a real category and a predicted result, respectively.

(33) The segmentation loss:
L.sub.segEntropy_Loss(y, y*)

(34) In step 400, the parameters of the 3D box include a center, a size, an angle and a corner.

(35) The step of estimating the parameters of the 3D box in the point cloud of interest based on the network with the weighted channel features specifically includes:

(36) Step 401: A center loss, a size classification loss, a residual size regression loss, an angle classification loss, a residual angle regression loss and a corner loss of the 3D box are calculated based on the network with the weighted channel features.

(37) The Hingle loss function:

(38) ${Hingle}_{δ} (x) = {\begin{matrix} 0.5 x^{2}, & .Math. x .Math. \leq δ \\ δ (.Math. x .Math. - 0.5 δ), & otherwise \end{matrix} .$

(39) (1) The center loss:
L.sub.center=Hingle.sub.67 =2.0(x).

(40) (2) The size classification loss of the 3D box:
L.sub.size-cls=Entropy_Loss (y, y*).

(41) (3) The residual size regression loss of the 3D box:
L.sub.size-reg=Hingle.sub.δ=1.0(x)

(42) (4) The angle classification loss of the 3D box:
L.sub.angle-clsEntropy_Loss(y, y*)

(43) (5) The residual angle regression loss of the 3D box:

(44) $L_{a n gle - r e g} = {.Math.}_{m}^{S} {.Math.}_{n}^{H} M a s k_{m n} [2 - 2 \cos (θ_{m n} - θ_{m n}^{*})];$

(45) wherein, S represents the number of size templates of the 3D box, m=1, 2, . . . S, H represents the number of direction angle templates of the 3D box, and n=1, 2, . . . , H; Mask.sub.mn represents a mask of an m.sup.th size template of the size templates of the 3D box and a mask of an n.sup.th direction angle template of the direction angle templates of the 3D box; θ.sub.mn represents a training parameter of a network in the m.sup.th size template of the 3D box and the n.sup.th direction angle template of the 3D box; and θ*.sub.mn represents a predicted parameter of the network in the m.sup.th size template of the 3D box and the n.sup.th direction angle template of the 3D box.

(46) (6) The corner loss of eight corners of the 3D box:

(47) 0 $L_{corner} = {.Math.}_{i = 1}^{S} {.Math.}_{j = 1}^{H} δ_{ij} \min {{.Math.}_{k = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{+} .Math., {.Math.}_{k = 1}^{8} .Math. P_{k}^{ij} - P_{k}^{-} .Math.};$

(48) wherein, P.sub.k.sup.ij represents a real coordinate of the j.sup.th direction angle template of the i.sup.th size template in the k.sup.th corner, P.sub.k.sup.+ represents a predicted coordinate of the k.sup.th corner, and P.sub.k.sup.− represents a coordinate of the corner after the 3D box is rotated by 180 degrees around a vertical direction; δ.sub.ij represents a real label of the j.sup.th direction template of the i.sup.th size template.

(49) Step 402: A total loss function is calculated according to the segmentation loss, the center loss, the size classification loss, the residual size regression loss, the angle classification loss, the residual angle regression loss and the corner loss of the 3D box.

(50) The total loss function L is calculated according to the following formula:
L=L.sub.seg+λ(L.sub.center+L.sub.size-reg+L.sub.size-cls+L.sub.angle-reg+L.sub.angle-cls)+γL.sub.corner;

(51) wherein, λ represents a weight of the center loss, angle loss and size loss of the 3D box, and γ represents a weight of the corner loss.

(52) Compared with the prior art, the present invention has the following significant advantages: 1) Since the method of the present invention uses Lidar and Image, which have a complementary effect, the method obtains significantly improved detection effect when performing detection on small objects. 2) The algorithm can directly perform detection on all categories of objects without the need of training different models for different categories, thus making the method especially versatile. 3) The weights among different channel features are considered in the algorithm to suppress unimportant features and enhance key features, thus extracting the features of the point cloud more effectively.

(53) The present invention further provides a three-dimensional object detection system based on weighted channel features of a point cloud, which can reduce interference in the point cloud.

(54) As shown in FIG. 2, the three-dimensional object detection system based on the weighted channel features of the point cloud according to the present invention includes the extraction unit 1, the determination unit 2, the segmentation unit 3 and the estimation unit 4.

(55) The extraction unit 1 is configured to extract a target in a two-dimensional image by a pre-trained deep convolutional neural network to obtain a plurality of target objects.

(56) The determination unit 2 is configured to determine a point cloud frustum in a corresponding three-dimensional point cloud space based on each target object.

(57) The segmentation unit 3 is configured to segment the point cloud in the frustum based on a point cloud segmentation network to obtain a point cloud of interest.

(58) The estimation unit 4 is configured to estimate parameters of a 3D box in the point cloud of interest based on a network with the weighted channel features to obtain the parameters of the 3D box for three-dimensional object detection.

(59) Compared with the prior art, the three-dimensional object detection system based on the weighted channel features of the point cloud according to the present invention has the same advantages as those of the above-mentioned three-dimensional object detection method based on the weighted channel features of the point cloud, and thus is not repeatedly described herein.

(60) Hereto, the technical solutions of the present invention have been described with reference to the preferred embodiments and the drawings. However, those skilled in the art can understand that the scope of protection of the present invention is obviously not limited to these embodiments. Those skilled in the art can make equivalent modifications or replacements to related technical features without departing from the principle of the present invention, and the technical solutions obtained by these modifications and replacements would fall within the scope of protection of the present invention.

Three-dimensional object detection method and system based on weighted channel features of a point cloud

Assignee

Inventors

Cpc classification

Classification Explorer

G06T7/11

PHYSICS

Classification Explorer

G06T15/005

PHYSICS

Classification Explorer

G06N3/08

PHYSICS

Classification Explorer

G06T7/97

PHYSICS

Classification Explorer

G06F18/00

PHYSICS

Classification Explorer

G06T7/10

PHYSICS

Classification Explorer

G06T2207/20084

PHYSICS

Classification Explorer

G06N3/045

PHYSICS

Classification Explorer

G06T2207/10028

PHYSICS

Classification Explorer

G06T2210/12

PHYSICS

Classification Explorer

G06V10/24

PHYSICS

International classification

Classification Explorer

G06T7/10

PHYSICS

Classification Explorer

G06N3/08

PHYSICS

Classification Explorer

G06T15/00

PHYSICS

Classification Explorer

G06T7/00

PHYSICS

Abstract

Claims

Description