LOCALIZING A FIFTH WHEEL HITCH COUPLER OF A MOTOR VEHICLE

Abstract

According to a computer-implemented method for localizing a fifth wheel hitch coupler (3) of a motor vehicle (1), a camera image (8) depicting the hitch coupler (3) is received from a camera (2) of the motor vehicle (1) and a top view image is generated by projecting the camera image (8) to a plane, which is perpendicular to a predefined height axis of the motor vehicle (1). A contour map (13) representing a contour (14) of a coupler throat (4) of the hitch coupler (3) is determined based on the top view image and a two-dimensional in-plane position of the coupler throat (4) is determined by fitting a predefined geometric figure (17) to the contour (14) of the coupler throat (4).

Claims

1. A computer-implemented method for localizing a fifth wheel hitch coupler of a motor vehicle, the method comprising: receiving a camera image depicting the fifth wheel hitch coupler from a camera of the motor vehicle; generating a top view image by projecting the camera image to a plane, which is perpendicular to a predefined height axis of the motor vehicle; determining a contour map representing a contour of a coupler throat of the fifth wheel hitch coupler based on the top view image; and determining a two-dimensional in-plane position of the coupler throat is determined by fitting a predefined geometric figure to the contour of the coupler throat.

2. The computer-implemented method according to claim 1, wherein prior to determining the in-plane position, the presence of the fifth wheel hitch coupler in the camera image is validated by applying an object detection algorithm to a predetermined region of interest within the camera image, inside which the fifth wheel hitch coupler is expected.

3. The computer-implemented method according to claim 1, wherein one of the preceding claims, generating the contour map comprises applying an edge detection algorithm to at least a part of the top view image.

4. The computer-implemented method according to claim 1, wherein the geometric figure comprises at least a part of a circle line.

5. The computer-implemented method according to claim 4, wherein fitting the geometric figure to the contour of the coupler throat comprises fitting a circle center of the circle line and/or fitting a circle radius of the circle line.

6. The computer-implemented method according to claim 5, wherein: for each of a plurality of predefined values for the circle radius, a scan area is determined depending on the contour of the coupler throat and the respective value for the circle radius, and fitting the geometric figure to the contour comprises varying the circle center within the respective scan area for each of the plurality of values for the circle radius.

7. The computer-implemented method according to claim 6, wherein: for each of the plurality of values for the circle radius and for each of a plurality of positions for the circle center within the respective scan area, at least one rating score is computed depending on the respective value for the circle radius and the respective position for circle center; and fitting the geometric figure to the contour comprises selecting one of the plurality of values for the circle radius and one of the plurality of positions for the circle center within the respective scan area depending on the at least one rating score.

8. The computer-implemented method according claim 7, wherein the at least one rating score comprises an intensity rating score, which depends on a sum of pixel values of the contour map for all pixel positions, which lie on the part of the respective circle line.

9. The computer-implemented method according to claim 8, wherein the at least one rating score comprises a symmetry rating score, which depends on a mirror symmetry of the contour of the coupler throat.

10. The computer-implemented method according to claim 9, wherein the at least one rating score comprises a combined score, which is given by a weighted sum of the intensity rating score and the symmetry rating score.

11. The computer-implemented method according to claim 10, wherein the one of the plurality of values for the circle radius and the one of the plurality of positions for the circle center within the respective scan area are selected, if at least one predefined condition is fulfilled, wherein the at least one condition is fulfilled only if: the respective combined score is greater than a predefined first threshold value; and/or the respective combined score is maximum for all of the plurality of values for the circle radius and all of the plurality of positions for the circle center; and/or the respective symmetry rating score is greater than a predefined second threshold value; and/or all pixel values of the contour map within a predefined environment of the respective circle line are smaller than a predefined third threshold value.

12. The computer-implemented method according to claim 1, wherein a height position of the coupler throat is determined depending on the in-plane position and depending on a predetermined height position of the camera.

13. A method for assisting coupling of a trailer with a fifth wheel hitch coupler of a motor vehicle, wherein: carrying out a computer-implemented method for localizing the fifth wheel hitch coupler of the motor vehicle according to claim 1; and guiding the motor vehicle at least in part automatically towards the trailer depending on a result of the localization of the coupler throat.

14. A system for localizing a fifth wheel hitch coupler of a motor vehicle, the system comprising: a camera for the motor vehicle, which is configured to generate a camera image depicting the fifth wheel hitch coupler; at least one computing unit, which is configured to: generate a top view image by projecting the camera image to a plane, which is perpendicular to a predefined height axis of the motor vehicle; determine a contour map representing a contour of a coupler throat of the fifth wheel hitch coupler based on the top view image; and determine a two-dimensional in-plane position of the coupler throat by fitting a predefined geometric figure to the contour of the coupler throat.

15. A computer program product comprising instructions, which, when executed by a data processing device, cause the data processing device to carry out a computer-implemented method according claim 1.

Description

[0082] In the figures,

[0083] FIG. 1 shows schematically a motor vehicle with a fifth wheel hitch coupler and an exemplary implementation of a system for localizing the fifth wheel hitch coupler according to the invention;

[0084] FIG. 2 shows the hitch coupler of the vehicle of FIG. 1 in more detail;

[0085] FIG. 3 shows a block diagram of an exemplary implementation of a computer-implemented method for localizing a fifth wheel hitch coupler according to the invention;

[0086] FIG. 4 shows an example of a contour map of a coupler throat;

[0087] FIG. 5 shows a further example of a contour map of a coupler throat; and

[0088] FIG. 6 shows a further example of a contour map of a coupler throat.

[0089] FIG. 1 shows schematically a motor vehicle 1 with a cargo bed 5 and a fifth wheel hitch coupler 3 mounted on a surface of the cargo bed 5 as well as an exemplary implementation of a system 7 for localizing the fifth wheel hitch coupler 3 according to the invention.

[0090] The system 7 comprises a camera 2 of the motor vehicle 1, in particular a rear-facing camera, wherein the cargo bed 5 is at least in part positioned in the field of view of the camera 2. For example, a so-called center high-mounted stop lamp camera, CHMSL-camera, may be used as the camera 2. The system 7 further comprises a computing unit 6, which is adapted to carry out an exemplary implementation of a computer-implemented method for localizing the fifth wheel hitch coupler 3 according to the invention.

[0091] FIG. 2 shows the hitch coupler 3 of the vehicle 1 in more detail. For example, the hitch coupler may contain a pedestal 18 mounted on the surface of the cargo bed 5 and a coupler component on top of the pedestal 18, which comprises a coupler throat 4. The coupler component with the coupler throat 4 may for example be essentially U-shaped. The coupler throat 4 may have the shape of a slit with an elongated portion and a rounded end into which a kingpin of a trailer (not shown) may be inserted to establish coupling between the motor vehicle 1 and the trailer.

[0092] According to the computer-implemented method, the computing unit 6 receives a camera image 8 from the camera 2 or a series of subsequent camera images of a camera image stream. The computing unit 6 generates a top view image by projecting the camera image 8 to a plane, which is perpendicular to a pre-defined height axis of the motor vehicle and determines a contour map 13 representing a contour 14 of the coupler throat 4 based on the top view image. The computing unit 6 determines a two-dimensional in-plane position of the coupler throat 4 by fitting a predefined geometric FIG. 17, for example a circle, to the contour 14 of the coupler throat 4.

[0093] In some implementations, the computing unit 6 determines a height position of the coupler throat 4 depending on the in-plane position of the coupler throat 4 and depending on a predetermined height position of the camera 2. The computing unit 6 may further use mechanical data of the motor vehicle, such as a mounting position of the coupler throat 4 with respect to a rear axle of the motor vehicle 1 and extrinsic calibration data 9b of the camera 2, in particular a mounting positon of the camera 2 with respect to the vehicle 1, for example the rear axle, in order to compute the height position.

[0094] Further details of the computer-implemented method are explained referring to specific implementations with reference to FIG. 3, FIG. 4, FIG. 5 and FIG. 6 in the following.

[0095] FIG. 3 shows a schematic block diagram of an algorithm for carrying out a computer-implemented method for localizing the fifth wheel hitch coupler 3 according to the invention.

[0096] Step S1 is optional and corresponds to a step for computing a region of interest, ROI. In this step, the ROI, where the hitch coupler 3 is expected in the camera image 8, is estimated depending on mechanical coupler data 9a of the hitch coupler 3, such as a predefined range for the height of the hitch coupler 3, a predefined range for a width and/or a predefined range for a length of the hitch coupler 3. Furthermore, for computing the ROI, mechanical vehicle data 9c, such as a center position of a rear axle of the vehicle 1 may be used.

[0097] In particular, the position of the hitch coupler 3 may be such that the center of the coupler throat 4 is positioned over a lateral center of the rear axle of the vehicle 1. Consequently, mechanical stability when coupling the trailer is improved. The information regarding center of the coupler throat 4 with respect to the rear axle may also be used for estimating the ROI. In implementations where step S1 is omitted, the whole camera image may be used instead of the ROI.

[0098] In a further optional step S2, an object detection algorithm may be applied to the ROI or the camera image in order to verify the presence of the hitch coupler 3 in the camera image 8. As a result of step S2, a first output 10 of the algorithm may be generated, which indicates whether or not the presence of the hitch coupler 3 in the camera image 8 has been verified. If this is not the case, said steps may be repeated for further camera images. On the other hand, if the presence of the hitch coupler 3 has been verified, the top view image may be generated in step S3 as described. Then, in step S4, the contour map 13 is generated and in step S5 the two-dimensional in-plane position of the coupler throat 4 is determined as described.

[0099] As a second output 11 of the algorithm or the method, respectively, the two-dimensional in-plane position and/or the height positon of the hitch coupler may be provided.

[0100] Further optional outputs 12, such as an algorithm status or a general status of the method may be provided, for example depending on whether an error has occurred, the algorithm is running or the algorithm has ended et cetera.

[0101] For verifying the presence of the hitch coupler 3 in step S2, for example a trained CNN, which is trained to specifically look for the presence of the hitch coupler 3 over the cargo bed 5 may be used. In case step S1 is carried out, the ROI may be given as an input to the CNN, otherwise the whole camera image 8 may be used. In case of a CMYK image, the CNN may for example use only the Y-channel to reduce the computational complexity. To be able to handle arbitrary image sizes, the images may be scaled to a fixed size to avoid multiple re-sizing and to provide a unified input to the CNN.

[0102] By generating the top view image in step S3, artifacts due to the camera perspective or caused by distortions may be removed or reduced. A height to set the top view plane may be given by the height position of the camera with respect to the cargo bed 5.

[0103] For generating the contour map 13 in step S4, an edge detection algorithm may be applied to the camera image 8 or, if applicable, to the ROI in the top view image.

[0104] In some implementations, generating the top view image may also comprise averaging the respective individual top view images of two or more camera images 8, in particular subsequent camera images 8. This may improve the performance under varying lighting conditions. After the averaging, the results may be Gaussian blurred in order to reduce noise.

[0105] The edges may be considered to consist of pixels at which a significant variation of the luminosity of the final top view image is present. Vertical and horizontal edges may be found by calculating X and Y components of the luminosity gradient, which may be calculated by using Sobel operators. Very high and very low intensity pixels may be filtered based on a gradient direction. All pixels that overcome a predefined threshold may be considered as belonging to an edge. However, also other kinds of edge detection algorithms may be used. The edges may also be labelled, wherein zero may be used for every pixel in the absence of an edge and all other connected edges may be labelled with a non-zero ID.

[0106] Step S5 may then use the contour 14 of the contour map 13, as shown in FIG. 4, FIG. 5 and FIG. 6 and a predefined plurality of circle radii 9d to localize the coupler throat 4. As shown by FIG. 4 to FIG. 6, a circle fit methodology may be used to find the center of the hitch coupler throat 4 by varying the radius to find the circle that best fits the contour 14.

[0107] For example, a Hough-transform technique may be used to find imperfect instances of circles within a contour by a voting procedure. For each radius and contour patch width, a respective scan area 15 may be build. The center position 16 of the circle 17 may be scanned within the scan area 15. This may be carried out for each of the plurality of circle radii 9d and for each combination of circle radius and center position 16, a rating score S may be computed.

[0108] FIG. 4 shows an example for the circle radius, which is significantly smaller than the radius of the contour 14, while in FIG. 5, the approximately correct radius is used. FIG. 6 shows two circles 17, 17 with the same radius but different center positions 16, 16 respectively.

[0109] The rating score S may for example be computed depending on an intensity rating score S.sub.i and a symmetry rating score S.sub.s. For example, S may be given as a weighted sum of S.sub.i and S.sub.s, for example S=0.8 S.sub.i+0.2 S.sub.s.

[0110] S.sub.i corresponds for example to the pixel integration over the circle arc of the current radius, wherein the angles may be restricted to a certain predefined angle or range. The sum may be normalized by a constant, which represents the maximum total intensity for the half circle of the respective radius, such that

[00001]$S_i=\frac{.Math.P_{R,C}\left({}_i\right)}{{\mathrm{RI}}_{\max }}$

[0111] This normalization permits to compare the intensity rating score S.sub.i over circle with different radii. The symmetry rating score S.sub.s measures the contour symmetry of the contour 14. The actual coupler throat 4 is expected to be mirror symmetric, so the best circle is expected to fit both sides of the coupler throat 4. The symmetry rating score S.sub.s may be given by the ratio between the sum of the left part and the total intensity sum, such that

[00002]$S_s=1-.Math.0.5-\frac{{.Math.}^{}{P}_{R,C}\left({}_i\right)}{.Math.P_{R,C}\left({}_i\right)}.Math.=1-.Math.0.5-\frac{S_{i,\mathrm{left}}}{S_i}.Math.$

[0112] The better the circle fits both sides, the closer the symmetry rating score S.sub.s is to 1. If the circle fits only one side, the symmetry rating score S.sub.s is approximately 0.5.

[0113] The best circle fit may for example fulfil the following conditions: [0114] 1. The score S must be higher than a predefined first threshold value [0115] 2. The symmetry rating score S.sub.s must be greater than a predefined second threshold value [0116] 3. The respective circle must not have particularly high intensity contour pixels in its immediate environment [0117] 4. The circle must have the maximum score S compared to all other circles.

[0118] For example, the height position of the coupler throat 4 may be estimated by a vector representation of the 3D line equation wherein the view part pixel is mapped to the camera pixel from which the world point is estimated by re-projecting the camera point onto a vertical plane. The position of the coupler throat 4 is estimated by using the vector representation of a line in 3D. This may be given by

[P]=[P0]+d*[D],

wherein [D] is the direction vector, [P0] is a reference point on the line, [P] is the resultant point and d a the scalar value of distance along the line from the reference point [P0].

[0119] Assuming the circle center of the hitch coupler throat 4 is at the rear axle with the known world ray X and reference camera position, the scalar value d may be computed. Hence, d=([X][X0])/[Dx], wherein [X] is an axial distance, [X0] is the x-position of the camera 2 and [Dx] the world ray x-component found by re-projection. Substituting d back to the equation, the world ray y-component and world ray z-component are found as [Y]=[Y0]d*[Dy] and [Z]=[Z0]d*[Dz].

[0120] The final result may be refined by means of different techniques in some implementations. For example, the camera 2 may provide a camera image 8 at each of subsequent frames. For each of the frames, the height position may be computed as described. A mean value may be computed by averaging the height over a plurality of frames and/or a medium value may be computed over a window of a plurality of frames and/or a mean absolute deviation, MAD, may be computed.

[0121] As described, in particular with respect to the figures, according to the invention the fifth wheel hitch coupler of a motor vehicle may be localized with improved accuracy and reliability.

[0122] 15 In several implementations, a way of confirming the presence of the coupler over the cargo bed is combined with the estimation of the height position of the hitch coupler. For example, once the presence of the hitch coupler has been verified, the hitch coupler may be localized further.

[0123] Advantages of the various implementations of the invention include that there are no dependencies on the actual implementation of the target to be detected and no dependency is on data to be provided by a driver of the motor vehicle. Localizing the position of the fifth wheel hitch coupler may be used for improved automatic hitching of trailer vehicles or for driver assistance.