Autonomous Driving System In The Agricultural Field By Means Of An Infrared Camera

Abstract

Autonomous driving method in the agricultural field by means of an thermal camera comprising the procedure of obtaining an interpolating function of at least two pixels, of alignments of plants or swaths of a thermal image that appears in front of an agricultural vehicle, acquired through at least one thermal camera, said at least two pixels being corresponding to at least two homologous peaks identified in as many at least two vectors built on values of temperature intensity of corresponding consecutive pixels belonging to as many straight and horizontal lines of pixels extracted from the thermal image and a procedure for calculating an angular phase shift and/or a lateral deviation of the interpolating function with respect to a vertical axis of the thermal image.

Claims

1. A method for autonomous driving in the agricultural field, comprising: obtaining an interpolating function of at least two pixels of alignments of plants or swaths of a thermal image that appears in front of an agricultural vehicle acquired through at least one thermal camera, wherein the at least two pixels correspond to at least two homologous peaks identified in at least two corresponding vectors built on values of temperature intensity of corresponding consecutive pixels belonging to straight and horizontal lines of pixels extracted from the thermal image; and determining an angular phase shift, a lateral deviation, or a combination thereof, of the interpolating function with respect to a vertical axis of the thermal image.

2. The method according to claim 1, wherein the thermal image is extracted so a relative medial axis is vertical and perpendicular to a longitudinal development axis of the agricultural vehicle.

3. The method according to claim 1, wherein at least one positive or negative peak in each of the at least two corresponding vectors is identified by means of a Gaussian or polynomial fitting procedure.

4. The method according to claim 1, wherein the thermal image has a rectangular shape, a horizontal orientation is identified coincidently at a base of the thermal image, and a vertical orientation is perpendicular to the base.

5. The method according to claim 1, wherein a positive peak in a second vector of the at least two corresponding vectors is homologous to a positive peak in a first vector of the at least two corresponding vectors when the positive peak in the second vector is contained in a neighborhood of the positive peak in the first vector transposed into the second vector.

6. The method according to claim 1, comprising searching for the at least two homologous peaks after windowing through a window which is approximately rectangular and central to the thermal image.

7. The method according to claim 5, wherein an amplitude of the neighborhood of the positive peak in the first vector is iteratively enlarged until the positive peak in the second vector is identified.

8. The method according to claim 1, comprising the following steps in succession: acquisition or extraction of the thermal image; extraction of the at least two lines of pixels; construction of the at least two corresponding vectors; Gaussian or polynomial fitting to get at least one positive or negative peak in each of the at least two corresponding vectors; identification of the at least two homologous peaks by transposition on a second vector of the at least two corresponding vectors of a neighborhood centered on a peak of the at least one positive or negative peak in a first vector of the at least two vectors; and interpolation through the interpolating function; wherein the vertical line is a medial axis of the thermal image or an axis of the ordinates when determining the angular phase shift, and the vertical line is the medial axis of the thermal image when determining the lateral deviation.

9. The method according to claim 1, comprising a procedure for selecting the straight and horizontal lines of pixels extracted from the thermal image, comprising: preliminary subdivision of the thermal image into two or more horizontal bands; extraction of at least two straight first horizontal lines of pixels belonging to different bands of the two or more horizontal bands; Gaussian or polynomial fitting to get at least one positive or negative peak in each of the at least two corresponding vectors and determining a relative fitting coefficient on each vector of the at least two corresponding vectors corresponding to each line of the at least two straight first horizontal lines, if the relative fitting coefficient is lower than a threshold then: selection of a straight second horizontal line of pixels belonging to the same horizontal band and different from a corresponding straight first horizontal line of the at least two straight first horizontal lines.

10. An autonomous driving system integrated into the agricultural vehicle equipped with a propulsion system, a seering system, and the at least one thermal camera, in which the autonomous driving system includes a processing unit configured to perform the method of claim 1.

11. The autonomous driving system according to claim 10, wherein the processing unit is configured to control the steering system, the propulsion system, or a combination thereof, so as to obtain an advancement of the agricultural vehicle parallel or superimposed with an alignment of plants or swaths.

12. Computer readable medium comprising program coding arranged to perform the method of claim 1 when the program coding is run on a computer.

13. (canceled)

14. The method according to claim 4, wherein the vertical orientation is coincident with one vertical side of the thermal image.

15. The method according to claim 1, wherein a negative peak in a second vector of the at least two corresponding vectors is homologous to a negative peak in a first vector of the at least two corresponding vectors when the negative peak in the second vector is contained in a neighborhood of the negative peak in the first vector transposed into the second vector.

16. The method according to claim 15, wherein an amplitude of the neighborhood of the negative peak in the first vector is iteratively enlarged until the negative peak in the second vector is identified.

Description

BRIEF DESCRIPTION OF FIGURES

[0033] Further objects and advantages of the present invention will become clear from the following detailed description of an example of its embodiment (and its variants) and from the annexed drawings given purely for explanatory and non-limiting purposes, in which:

[0034] FIG. 1 shows a side view of an example of an agricultural vehicle equipped with a thermal imaging camera;

[0035] FIG. 2 shows a thermal image captured or extracted from the scenario, captured by the thermal camera which faces the vehicle of FIG. 1;

[0036] FIG. 3 shows two temperature vectors of as many corresponding lines of pixels extracted from the image of FIG. 2;

[0037] FIG. 4 shows the interpolation of maximum homologues relating to the two vectors in FIG. 3, identifying an angular phase shift between the vehicle's development axis and an alignment of plants or swaths;

[0038] FIG. 5 shows the interpolation of maximum homologues relating to the two vectors in FIG. 3, identifying a linear displacement between the development axis of the vehicle and an alignment of plants or swaths;

[0039] FIG. 6 shows a flow diagram of an example of the method object of the present invention,

[0040] FIG. 7 shows the fitting procedure operated on curves constructed on the temperature intensities of the vectors of FIG. 3 and the interpolation of the respective central maxima.

[0041] The same reference numbers and letters in the figures identify the same elements or components.

[0042] In the context of the present description, the term “second” component does not imply the presence of a “first” component. These terms are in fact used as labels to improve clarity and should not be understood in a limiting way.

[0043] The elements and features illustrated in the various preferred embodiments, including the drawings, can be combined with each other without however departing from the scope of this application as described below.

DETAILED DESCRIPTION OF IMPLEMENTATION EMBODIMENTS

[0044] With reference to FIG. 1, where an AV agricultural vehicle is shown to which at least one thermal camera is associated. The thermal camera can be positioned so that a relative vision axis is parallel with a development axis of the vehicle.

[0045] According to a top view of the vehicle, the development axis of the vehicle and the viewing axis of the camera are at a fixed distance that can be zero. Since the camera's viewing axis is confused with the camera itself, then the expression “camera parallel to” means that “the camera's viewing axis is parallel to”. Furthermore, when we say that the camera is “aligned with the development axis of the vehicle” it means that the aforementioned distance is zero.

[0046] It is worth highlighting that it is not an important aspect the alignment of one or more thermal imaging cameras with the vehicle's development axis as the fact that an approximately rectangular thermal image that appears in front of the vehicle is extracted. In fact, thermal imaging cameras can also be angled to widen the horizon of vision that lies ahead of the vehicle for further purposes.

[0047] For example, it is possible to provide a plurality of thermal imaging cameras arranged in a radial pattern to have a 360° thermal view around the vehicle.

[0048] An example of an extracted image is shown in FIG. 2. Of the extracted image, each pixel represents an absolute or relative temperature of the corresponding object that is present in the scene.

[0049] Preferably, at least one thermal camera acquires a temperature image, which is calculated by means of a calibration carried out in advance and a definition of the emissivity coefficient of the objects in the scene which in this case are organic samples.

[0050] In other words, the autonomous driving system, in relation to the type of mission, can apply different calibrations to the thermal imaging camera.

[0051] According to the present invention, from the thermal image acquired or extracted from a larger image, at least two lines of pixels parallel to each other and horizontal are selected, that is, parallel with the abscissa axis X associated with the thermal image.

[0052] In FIG. 2, three lines of pixels are extracted: Line 1, Line 2, Line 3.

[0053] Increasing the number of extracted lines increases the accuracy of the vehicle alignment with the alignment of plants or swaths.

[0054] From each line, the temperature intensity associated with each pixel of the same line is extrapolated and a vector V1, V2, etc . . . of temperature values corresponding to the pixels of the same line is generated.

[0055] A Gaussian or other fitting is performed on each vector—polynomial, catenary, etc . . . identifying one or more maxima: M1, M2, M3.

[0056] For convenience, the notation V1:M1 is used to indicate that the first maximum M1 belongs to the first vector V1, etc . . .

[0057] Obviously, if the thermal camera is placed very high with respect to the plant alignments, the thermal image also contains information about the lateral alignments. Therefore, it is likely that in each vector as many maxima are identified as the alignments visible in the thermal image.

[0058] The same fitting operation is performed on the other vectors, V2, V3, etc . . .

[0059] The so-called homologous maxima V1:M1-V2:M1, V1:M2-V2:M2 etc are identified, i.e. those maxima, within the two or more vectors that are in positions, within their own vector, similar to the position of the other maxima in their respective vectors.

[0060] This is equivalent to selecting a first maximum of a first vector, identifying a neighborhood I1 of the first maximum of the first vector V1 and identifying a first maximum M1 of the second vector V2, within the same neighborhood I1 transposed on the second vector.

[0061] If it is not possible to identify the maximum homologue, it proceeds to widen the neighborhood of the first maximum and therefore its transposition on the second vector.

[0062] For calculation economy, it is advisable to discard the lateral maxima by windowing the original thermal image.

[0063] In other words, preferably, before extracting the pixel lines, constructing the temperature vectors, identifying the maximums, it proceeds to apply an approximately central W window, with respect to the vehicle development axis.

[0064] FIGS. 4 and 5 show the window W and the vehicle development axis which may not coincide with the medial vertical axis of the thermal image. In any case, this is a transposition within the tasks of the person skilled in the art.

[0065] Once the homologous maxima have been identified, it proceeds to calculate an interpolation line r which connects the homologous maxima, preferably only the central ones, i.e. those falling in the window W.

[0066] At least during the phase of calculating the interpolation line, the homologous maxima are placed in the position of the corresponding pixels. In other words, the interpolation is performed between the pixels of the thermal image corresponding to the maximum homologues.

[0067] When the extracted lines are two, then it is possible to speak of an interpolating line, otherwise it is more correct to speak of an interpolating function because it is unlikely that the three or more homologous maxima will be perfectly aligned.

[0068] The central axis Z of the window is an axis perpendicular to the development axis of the vehicle and parallel to the Y axis associated with the thermal figure and the interpolation scheme of the maxima of vectors V1, V2, etc . . . of FIGS. 3-5.

[0069] FIG. 4 shows a condition in which the interpolating line r is angularly out of phase with the ordinate axis Y. In this case, the angle Alpha formed represents the angle that the autonomous driving system must recover to arrange the agricultural vehicle aligned with the alignment detected by thermal imaging camera.

[0070] FIG. 5 shows a condition in which the interpolating line r is laterally offset by a lateral offset Delta from the central axis Z. In this case, the autonomous driving system can ensure that the vehicle's axis Z remains at the predetermined distance Delta from an alignment or it can cause the vehicle to move so that the straight line r is superimposed on the central axis Z (Delta=0).

[0071] Obviously, the present invention can be limited to monitoring a misalignment of the vehicle with respect to an alignment of a plantation or swath, or, on the basis of the monitoring system, it can control the steering and advancement system of the vehicle to control its moving forward.

[0072] FIG. 6 describes an exemplary flow chart of the method object of the present invention. The method includes the following steps:

[0073] Step 1: capture or extract a thermal image,

[0074] Step 2: extraction of at least two lines of pixels Line 1, Line 2, etc . . . , i.e. straight horizontal lines,

[0075] Step 3: construction of as many vectors, in which in each position of one of the vectors there is a temperature intensity value of a corresponding pixel of the corresponding line,

[0076] Step 4: Gaussian or polynomial interpolation, etc. To obtain maxima M1, M2, M3, positive or negative on each of the vectors,

[0077] Step 5: identification of homologous maxima by transposition, on a second vector V2, of a neighborhood I1, I2, I3 centered on the maximum M1:V1, M2:V1, M3:V1 of a first vector, and possibly repetition of this step to all other vectors with respect to the first two,

[0078] Step 6: interpolation by interpolating line r of homologous maxima, in particular of only the maxima contained in a central window W of the thermal image,

[0079] Step 7: determination of an Alpha angle of phase shift of the agricultural vehicle with an alignment of plants or swaths, in which said angle is given by the intersection between the medial Z axis of the thermal image or the Y, Y′ ordinate axis and the interpolating line r, e/o

[0080] Step 7bis: determination of a lateral Delta offset of the agricultural vehicle with an alignment of plants or swaths, given by a distance between said interpolating straight line r and said medial axis Z of the thermal image.

[0081] Obviously, while the shifting angle is in itself metric, the lateral shift or offset can be calculated by reporting the distance in pixels in metric distance. This is easily achieved by knowing the characteristics of the camera's optics.

[0082] FIG. 6 indicates a dashed arrow, which takes into account the fact that more than two temperature intensity vectors can be constructed.

[0083] Since it is constructively imposed that the medial axis Z is parallel to the ordinate axis Y—and to any axis parallel Y′ to the ordinate axis Y—then the angular phase shift can be identified between the interpolating line r and any of Z, Y, Y′.

[0084] FIG. 7 graphically shows the fitting operations performed on line 1 and line 2. The maxima are interpolated and the identified line r is parallel with the medial axis Z, defining a lateral deviation Delta, which must be cancelled or maintained or increased, depending on the mission that the agricultural vehicle is facing.

[0085] The windowing operation W is indicated in a dotted block that is optional and can be performed at any point in the method.

[0086] It is worth noting that the transposition of the neighborhoods Il, I2, I3 may not be performed when the windowing is performed in the early stages of the method or when the thermal image is originally very narrow, in terms of the field of view.

[0087] According to a preferred variant of the invention, also the lateral maxima M1, M3 on the vectors V1, V2, etc . . . are taken into consideration when it may happen that the reference alignment is interrupted for any reason. In this case, any angular errors due to optical aberrations are compensated by considering at the same time a right interpolating line and a left interpolating line. These compensation operations are within the tasks of the skilled person in the art who knows the laws of optics.

[0088] Regarding the selection of pixel lines, a preferred method is indicated below.

[0089] Meanwhile, it is worth pointing out that the consistency between the calculated interpolating function and the alignment of the plants or swaths is better as the more uniform is the distribution of pixel lines Line 1, Line 2, etc . . . in the thermal image.

[0090] Preferably, everything above the horizon is masked, i.e. eliminated, from the thermal image.

[0091] The remaining portion of the image is divided into horizontal bands, and a line of pixels is extracted within each band.

[0092] When the fitting operation is performed, it is possible to discard a pixel line in favor of another pixel line belonging to the same band, when a correlation coefficient (Pearson), relative to the fitting procedure, is below a predetermined threshold. Therefore, the lines can be selected in order to have an optimal Gaussian fitting.

[0093] This means that steps 2 to 4 can be looped through to an optimal situation.

[0094] Considering today's computing capabilities, this can be done while the vehicle is moving forward, without any particular risk.

[0095] The present invention can be advantageously realized by means of a computer program, which comprises coding means for carrying out one or more steps of the method, when this program is executed on a computer. Therefore, it is intended that the scope of protection extends to said computer program and further to computer readable means comprising a recorded message, said computer readable means comprising program coding means for carrying out one or more steps of the method, when said program is run on a computer.

[0096] Implementation variants of the described non-limiting example are possible, without however departing from the scope of protection of the present invention, including all the equivalent embodiments for a person skilled in the art, to the content of the claims.

[0097] From the above description, the person skilled in the art is able to realize the object of the invention without introducing further construction details.