Target detection and tracking method
10909689 ยท 2021-02-02
Assignee
Inventors
Cpc classification
G08B29/185
PHYSICS
International classification
Abstract
A method is provided for detecting and tracking targets in a series of images, at least one track as the existing track, being formed from spots detected in images in the series preceding a current image. The method includes, for the current image: obtaining at least one spot included in the current image and, for each spot, a value representing a characteristic of the spot; classifying each spot according to the representative value in a first category when the representative value is higher than a first predefined threshold or in a second category when the representative value lies between the first predefined threshold and a lower second predefined threshold; for each existing track, allocating a spot in the first category to the existing track and allocating a spot in the second category to the existing track when no spot in the first category is compatible with said existing track.
Claims
1. A method for detecting and tracking targets in a series of images supplied by an optronic system, wherein the method comprises, for a current image: obtaining at least one spot included in said current image and, for each spot, a value representing a characteristic of said spot; classifying each spot according to said representative value in a first category when said representative value is higher than a first predefined threshold in a second category when said representative value lies between the first predefined threshold and a second predefined threshold lower than the first predefined threshold, the spots in the second category having a probability of false alarm higher than the spots in the first category; for each existing spatiotemporal chain of spots of a first type, referred to as an existing track of the first type, which has been formed from spots detected in images in said series preceding the current image, allocating a spot in the first category to said existing track of the first type when at least one spot in the first category is compatible, in accordance with a predefined criterion, with said existing track of the first type and, when no spot in the first category is compatible with said existing track of the first type, allocating a spot in the second category when at least one spot in the second category is compatible, in accordance with the predefined criterion, with the existing track of the first type, an allocation of an existing track of the first type making it possible to pursue said existing track of the first type in the current image; creating a new track of the first type for each spot in the first category that has not been allocated to an existing track; classifying each spot in the second category that has not been allocated to an existing track of the first type in a third category if a number of pixels making up this spot is higher than a third predefined threshold and in a fourth category otherwise; for each existing spatiotemporal chain of spots of at least one second type, referred to as an existing track of the second type, which has been formed from spots detected in images in said series preceding the current image, allocating a spot in the third category to said existing track of the second type when at least one spot in the third category is compatible, in accordance with the predefined criterion, with said existing track of the second type and transforming said existing track of the second type into an existing track of the first type when the number of spots making up said track exceeds a predefined number of spots; creating a new track of the second type for each spot in the third category that has not been allocated to an existing track of the second type.
2. The method according to claim 1, wherein the value representing a characteristic of said spot is an snr value representing snr values of pixels belonging to said spot, the first and second predefined thresholds being of predefined snr values.
3. The method according to claim 1, wherein a spot is compatible with an existing track of the first or second type when it is included in a subpart of the current image, referred to as a validation window, corresponding to said track, a validation window being defined around a prediction of a position, referred to as a predicted position, in which a spot corresponding to said track in the current image must be situated, said position being obtained from a track maintenance function taking into account kinematic properties of a target tracked using said track.
4. The method according to claim 3, wherein, when a plurality of spots in the first category are compatible with an existing track of the first type, the spot in the first category closest to the predicted position in accordance with a predefined proximity criterion is allocated to said track and, when a plurality of spots in the second category are compatible with an existing track of the first type, the spot in the second category closest to the predicted position according to the predefined proximity criterion is selected, only spots in the first or second category being able to be allocated to a track of the first type.
5. The method according to claim 3, wherein, when a plurality of spots in the third category are compatible with an existing track of the second type, the spot in the third category closest to the predicted position in accordance with the predefined proximity criterion is allocated to said track, only spots in the third category being able to be allocated to a track of the second type.
6. The method according to claim 1, wherein the tracks of the second type comprise two track subtypes, a track of the second type being of the first subtype as long as it comprises a number of spots of the third category smaller than a predefined number of spots, a track of the second type being of the second subtype when it comprises a number of spots greater than the first predefined number of spots, a track of the second type of the second subtype being transformed into a track of the first type when a sum of snr values representing spots constituting said track of the second type is higher than a fourth predefined threshold over a predefined number of successive spots, or when a change in the size of the spots of said track of the second type is strictly increasing over a second predefined number of successive spots, or when a change in the snr of the spots of said track of the second type is strictly increasing over a third predefined number of successive spots.
7. The method according to claim 6, wherein, following the allocation of the spots of the first or second categories to the existing track of the first type, and before the creation of a new track of the first type for each spot in the first category that has not been allocated to an existing track, the method further comprises, for each existing track of the second type of the second subtype, allocating a spot in the first category to said track when at least one spot in the first category is compatible, in accordance with a predefined allocation criterion, with said track and, when no spot in the first category is compatible with said track, allocating a spot in the second category when at least one spot in the second category is compatible, in accordance with the predefined allocation, with said track.
8. A device for detecting and tracking targets in a series of images supplied by an optronic system, wherein the device comprises electronic circuitry configured to: obtain at least one spot included in a current image and, for each spot, a value representing a characteristic of said spot; classify each spot according to said representative value in a first category when said representative value is higher than a first predefined threshold in a second category when said representative value lies between the first predefined threshold and a second predefined threshold lower than the first predefined threshold, the spots in the second category having a probability of false alarm higher than the spots in the first category; for each existing spatiotemporal chain of spots of a first type, referred to as an existing track of the first type, which has been formed from spots detected in images in said series preceding the current image, allocate a spot in the first category to said existing track of the first type when at least one spot in the first category is compatible, in accordance with a predefined criterion, with said existing track of the first type and, when no spot in the first category is compatible with said existing track of the first type, allocate a spot in the second category when at least one spot in the second category is compatible, in accordance with the predefined criterion, with the existing track of the first type, an allocation of an existing track of the first type making it possible to pursue said existing track of the first type in the current image; create a new track of the first type for each spot in the first category that has not been allocated to an existing track; classify each spot in the second category that has not been allocated to an existing track of the first type in a third category if a number of pixels making up this spot is higher than a third predefined threshold and in a fourth category otherwise; for each existing spatiotemporal chain of spots of at least one second type, referred to as an existing track of the second type, which has been formed from spots detected in images in said series preceding the current image, allocate a spot in the third category to said existing track of the second type when at least one spot in the third category is compatible, in accordance with the predefined criterion, with said existing track of the second type and transform said existing track of the second type into an existing track of the first type when the number of spots making up said track exceeds a predefined number of spots; create a new track of the second type for each spot in the third category that has not been allocated to an existing track of the second type.
9. An equipment comprising a device according to claim 8.
10. A non-transitory storage medium storing a computer program comprising instructions for implementing, by a device, the method according to claim 1, when said program is executed by a processor of said device.
Description
(1) The features of the invention mentioned above, as well as others, will emerge more clearly from a reading of the following description of an example embodiment, said description being given in relation to the accompanying drawings, among which:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9) The invention is described hereinafter in a context of air surveillance using an image acquisition device making acquisitions of images in a visible domain. The invention could just as well apply in different surveillance contexts such as maritime surveillance. Moreover, the invention applies to other types of image (i.e. other spectral domains) such as infrared images, thermal images and radar images.
(10)
(11) The equipment 10 comprises an image acquisition module 100 such as a camera operating in the visible spectral domain. The image acquisition module 100 supplies the images that it has acquired to an image processing module 101. The image processing module 101 is able, among other things, to detect spots in each image and to calculate at least one value representing each spot.
(12) The image processing module 101 supplies the acquired images, the position of each spot detected and, for each spot detected, a value representing said spot to an analysis module 102. The analysis module 102 implements a target detection and monitoring method according to the invention. This method, which we describe hereinafter in relation to
(13) For each image acquired, the analysis module 102 transmits an item of information representing each track confirmed to a display module 103, which displays this information, so that it can be used by an operator. For example, the analysis module 102 supplies to the display module 103 information representing a position of each spot detected in the last image acquired which was able to be allocated to a confirmed track.
(14) It is considered that the equipment 10 forms an optronic system.
(15) In
(16)
(17) According to the example of hardware architecture depicted in
(18) The processor 1021 is capable of executing instructions loaded in the RAM 1022 from the ROM 1023, from an external memory (not shown), from a storage medium (such as an SD card), or from a communication network. When the equipment 10 is powered up, the processor 1021 is capable of reading instructions from the RAM 1022 and executing them. These instructions form a computer program causing the complete or partial implementation, by the processor 1021, of the methods described hereinafter in relation to
(19) The methods described in relation to
(20)
(21) The method is implemented for each image acquired by the image acquisition module 100. An image being processed by the equipment 10 is hereinafter referred to as the current image. In one embodiment, we assume that at least one track, referred to as the existing track, has been formed from spots detected in images preceding the current image.
(22) As we shall see hereinafter, the method uses a plurality of types of track comprising tracks of the nominal type, referred to as nominal tracks, and at least one second type of track comprising tracks of the TBD type, referred to as TBD tracks (track before detect).
(23) A track is considered to be of the nominal type when it consists of a spot or a set of spots that are of interest in performing target monitoring in accordance with at least one first predefined condition. A first example of a first predefined condition is such that a track comprising at least one spot having an SNR value higher than a predefined SNR threshold is considered to be a nominal track. A second example of a first predefined condition is such that a track comprising a number of spots greater than a predefined number is considered to be a nominal track. Each nominal track may be a confirmed track, a track in the course of initialisation or a bifurcated track created from a confirmed track.
(24) A track is considered to be of the TBD type when it fulfils no first predefined condition. A TBD track must however fulfil at least one second predefined condition, which we describe below.
(25) It should be noted that only the nominal tracks can be confirmed by the analysis module 102. To do this, the analysis module 102 performs a track-management function consisting of confirming a nominal track when said nominal track has been constructed from Q spots found in Q images in a series of P consecutive images (P>Q), P and Q being predefined positive integer numbers. In one embodiment P=20 and Q=16. A track of the second type is never confirmed.
(26) In a step 30, the analysis module 102 obtains at least one spot included in said current image and, for each spot, a value representing a characteristic of said spot VR. In one embodiment, the value representing a spot VR is for example a mean SNR value of the pixels constituting said spot.
(27) In a step 31, the analysis module 102 classifies each spot according to said representative value VR in a first category when said value VR is higher than a first predefined threshold T.sub.1 or in a second category when said representative value VR lies between the first predefined threshold T.sub.1 and a second predefined threshold T.sub.2 lower than the first predefined threshold T.sub.1. The first category comprises spots for which the probability of false alarm is low, referred to as standby spots. The second category comprises spots for which the probability of false alarm is higher than for the standby spots. These spots classified in the second category are referred to as specific spots. In the embodiment in which the value representing a spot VR is a mean SNR value, the first and second thresholds T.sub.1 and T.sub.2 are SNR values obtained on training image sequences. We detail below the classification of the spots in categories in relation to
(28) In a step 32, the analysis module 102 uses an algorithm for allocation of spots to the existing nominal tracks.
(29) In the context of this algorithm, for each existing nominal track, the analysis module 102 allocates a standby spot to said existing nominal track when at least one standby spot is compatible, according to a predefined allocation criterion, with said existing nominal track. When no standby spot is compatible with said existing nominal track, the analysis module 102 allocates a specific spot when at least one specific spot is compatible, according to the predefined allocation criterion, with the existing nominal track.
(30) In this way the allocations of standby spots are privileged since these are the spots that have the least risk of false alarms (i.e. the least risk of generating false nominal tracks or pursuing nominal tracks whereas the corresponding target is no longer being observed by the image acquisition module 100). However, the specific spots are taken into account for pursuing existing nominal tracks for which no compatible standby spot has been found. Thus nominal tracks are pursued with spots of less good quality in terms of probability of false alarm. Nevertheless, using specific spots for pursuing an existing nominal track makes it possible to pursue nominal tracks that would have been abandoned in error, when, for example, a target is passing through an environment where detection is difficult or when the target is situated at the limit of range of the image acquisition module 100.
(31) It should be noted that, when no existing nominal track has been formed on the previous images, the step 32 of allocation of a track is not performed. The analysis module 102 then passes directly from the step 31 to a step 33.
(32) In the step 33, the analysis module 102 checks whether standby spots have not been allocated to an existing track.
(33) If such is the case, the step 33 is followed by a step 34, during which the analysis module 102 creates a candidate track from the standby spots that have not been allocated to an existing track.
(34) Following the step 34 (or following the step 33 if there no longer remain any standby spots after the use of the spot allocation algorithm), the analysis module 102 performs a step 35 during which the analysis module 102 determines whether at least one specific spot has not been allocated to an existing nominal track.
(35) If each specific spot has been allocated, the analysis module 102 awaits a new image in a step 45 and returns to the step 30 when a new image is supplied to it by the image processing module 101.
(36) If at least one specific spot has not been allocated, the analysis module 102 determines, for each remaining specific spot, whether this specific spot must be definitively rejected, or whether this spot must be preserved in order to create or continue a TBD track. To do this, the analysis module 102 performs the steps 36 to 39 for each specific spot not allocated. During the step 36, the analysis module 102 classifies said specific spot in a third category during a step 37 if a number of pixels making up this specific spot is higher than a threshold T.sub.3 and in a fourth category otherwise during a step 38. A spot in the third category is referred to as a TBD spot. The fourth-category spots are definitively rejected by the analysis module 102. In one embodiment T.sub.3=6 pixels.
(37) In a step 39, the analysis module 102 determines whether at least one other unallocated specific spot remains to be processed. If so, the analysis module 102 returns to the step 36. Otherwise the analysis module 102 performs a step 40.
(38) During the step 40, the analysis module 102 uses an algorithm for allocation of spots to the existing TDB tracks. In the context of this algorithm, for each existing TBD track, the analysis module 102 allocates a TBD spot to said existing TBD track when at least one TBD spot is compatible, according to the predefined allocation criterion, with said existing TBD track. The step 40 is detailed in relation to
(39) An example of a second predefined condition that must be complied with by a TBD track is therefore that it consists of TBD spots, that is to say consisting of at least one specific spot not allocated to an existing nominal track, having a representative value VR of less than T.sub.1 and greater than T.sub.2 and consisting of a number of pixels higher than the threshold T.sub.3.
(40) In a step 41, the analysis module 102 determines, for each existing TBD track, whether it can transform said TBD track into a nominal track. In one embodiment, the analysis module 102 transforms the TBD track into a nominal track when the number of spots making up said TBD track exceeds a predefined number NB1. In one embodiment NB1=40.
(41) In a step 43, the analysis module 102 determines whether TBD spots have not been allocated to an existing track. If all the TBD spots have been allocated, the analysis module 102 executes the step 45 already explained. Otherwise, in a step 44, the analysis module 102 creates a new TBD track for each TBD spot that has not been allocated to an existing track. The step 44 is followed by the step 45.
(42) In one embodiment, the tracks of the second type comprise the TBD tracks and so-called transient tracks. It can then be considered that the second type of track comprises two subtypes. The transient tracks are tracks intermediate between the TBD tracks and the nominal tracks. The analysis module 102 transforms a TBD track into a transient track when the number of spots making up said TBD track exceeds a predefined number NB2<NB1. In one embodiment NB2=20. The analysis module 102 transforms a transient track into a nominal track when a condition for transient to nominal transformation is complied with. For example, the analysis module 102 transforms a transient track into a nominal track when a sum of the mean SNRs of spots constituting the transient track is higher than a threshold T.sub.4 over a predefined number of successive spots nbp1, or when a change in the size of the spots of the transient track is strictly increasing over a predefined number of successive spots nbp2, or when a change in the SNR of the spots of the transient track is strictly increasing over a predefined number of successive spots nbp3. In one embodiment nbp1=nbp2=nbp3=10 and T.sub.4=30 db.
(43) In the embodiment comprising transient tracks, in step 32, the analysis module 102 uses the algorithm for allocation of spots to the existing nominal and transient tracks.
(44) The application of the algorithm for allocating spots to the existing nominal tracks is identical to the embodiment without a transient track described in relation to the step 32. When the existing nominal tracks have been run through, the analysis module 102 applies this algorithm to the existing transient tracks. Thus, for each existing transient track, the analysis module 102 allocates a standby spot to said existing transient track when at least one standby spot is compatible, according to a predefined allocation criterion, with said existing transient track. When no standby spot is compatible with said existing transient track, the analysis module 102 allocates a specific spot when at least one specific spot is compatible, according to the predefined allocation criterion, with the existing transient track.
(45) Thus standby or specific spots can be allocated to the transient tracks.
(46) In the embodiment comprising transient tracks, in step 40, for each existing TBD or transient track, the analysis module 102 allocates a TBD spot to said existing TBD or transient track when at least one TBD spot is compatible, according to the predefined allocation criterion, with said existing TBD or transient track.
(47) In the step 41, the analysis module 102 transforms a TBD track into a transient track when the number of spots making up said TBD track exceeds the predefined number NB2.
(48) In a step 42, the analysis module 102 transforms a transient track into a nominal track when the condition for transformation of transient to nominal is met.
(49) The steps 43, 44 remain identical.
(50) In one embodiment, the value representing a spot VR is a spot size in number of pixels constituting the spot, an entropy of the pixels constituting the spot, or a SURF descriptor (Speeded Up Robust Features, H. Bay et al., SURF: Speeded Up Robust Features, Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346-359, 2008).
(51) In the case of other types of value representing a spot VR (dimension of the spot, entropy of the pixels constituting the spot, SURF descriptor), the thresholds used T.sub.1, T.sub.2 and T.sub.3 are adapted to the representative values used.
(52) Using a plurality of types of track makes it possible to apply a different processing to each type of track. For example, a nominal track, if it is confirmed, is intended to be displayed on the display module 103. A TBD track is never displayed since, compared with a nominal track, the spots that constitute a TBD track have more chances of being false alarms. However, the more a TBD track persists, the more the probability of this track consisting of spots corresponding to false alarms decreases. This track may for example correspond to an object (for example an aircraft) remaining distant from the image acquisition module 100 as from a certain time, or an object that is difficult for the image processing module 101 to detect since it is well camouflaged. Under these conditions, it is preferable to indicate the presence of this object to an operator looking at the display module 103 and therefore to transform a TBD track into a nominal track.
(53)
(54) In a step 310, the analysis module 102 initialises a variable n to the value 0. The value n serves to count the spots to be processed during the step 31. At each iteration of the method described in relation to
(55) In a step 311, the analysis module 102 compares the value representing the spot Plot.sub.n, denoted VR.sub.n, with the first threshold T.sub.1. When VR.sub.nT.sub.1, the analysis module 102 classifies the spot Plot.sub.n in the standby spot category in a step 315.
(56) If VR.sub.n<T.sub.1, the step 311 is followed by a step 312 during which the analysis module 102 compares the value VR.sub.n with the second threshold T.sub.2. When VR.sub.nT.sub.2, the analysis module 102 classifies the spot Plot.sub.n in the specific spot category in a step 314.
(57) If VR.sub.n<T.sub.2, the step 312 is followed by a step 313 during which the spot Plot.sub.n is rejected by the analysis module 102. The spot Plot.sub.n is then not used for extending a track or for creating a candidate track.
(58) The steps 315 and 314 are followed by a step 316 during which the analysis module 102 increments by one unit the value of the variable n.
(59) In a step 317, the value of the variable n is compared with a variable N representing a number of spots detected by the image processing module 101. If n<N, the method described in relation to
(60) In one embodiment, when the value representing a spot is a mean SNR of the pixels constituting this spot, T.sub.1=12 dB and T.sub.2=8 dB.
(61) In one embodiment, when the value representing a spot is a mean SNR of the pixels constituting this spot, the threshold T.sub.1 is a predefined SNR value, which can be updated during the analysis of the series of images acquired according to mean SNR values of spots found in the images. The threshold T.sub.2 may be a relative value with respect to the threshold T.sub.1. For example, T.sub.2=T.sub.14 dB.
(62)
(63) In a step 320, the analysis module 102 performs a precorrelation between each spot classified in the step 31 and each existing nominal track. One objective of this precorrelation is to determine, for each existing nominal track, which spot could correspond to this track. In one embodiment, a track maintenance function is applied to each existing nominal track in order to determine where a spot corresponding to said track should be situated. This track maintenance function makes it possible to determine a position of a predicted spot and a validation window around said predicted spot. Each spot appearing in the validation window of an existing nominal track is a candidate for the association with said track.
(64) In a step 321, the analysis module 102 initialises a variable k to 0. The variable makes it possible to enumerate the existing nominal tracks.
(65) In a step 322, the analysis module 102 begins an analysis of the k.sup.th existing nominal track, denoted track.sub.k. In the step 322, the analysis module 102 determines whether at least one standby spot corresponds to the existing nominal track track.sub.k. If this is the case, in one embodiment, the analysis module 102, in a step 326, allocates the standby spot that is closest in accordance with a predefined metric to the position of the predicted spot corresponding to the existing nominal track track.sub.k. The predefined metric is for example a Euclidian distance.
(66) Otherwise, if no standby spot corresponds to the existing nominal track track.sub.k, in a step 323 the analysis module 102 determines whether at least one specific spot corresponds to the existing nominal track track.sub.k. If such is the case, in one embodiment, the analysis module 102, in a step 325, allocates the specific spot that is closest in accordance with the predefined metric to the position of the predicted spot corresponding to the existing nominal track track.sub.k.
(67) Otherwise, if no specific spot corresponds to the existing nominal track track.sub.k, the analysis module 102 ends the analysis of the track track.sub.k in a step 324.
(68) The steps 326, 323 and 324 are followed by a step 327, during which the analysis module 102 increments by one unit the variable k.
(69) In a step 328, the analysis module 102 compares the value of the variable k with a variable K corresponding to the number of existing nominal tracks. If the variable k equals the variable K, the analysis module 102 ends the step 32. Otherwise the analysis module 102 once again performs the step 322.
(70) In one embodiment, when an existing nominal track is not updated during a predefined number of images, said track is abandoned by the analysis module 102.
(71)
(72) In a step 400, the analysis module 102 performs a precorrelation between each TBD classified spot in step 37 and each TBD or transient track. One objective of this precorrelation is to determine, for each existing TBD or transient track, which spot could correspond to this track. In one embodiment, a track maintenance function is applied to each existing TBD or transient track in order to determine where a spot corresponding to said track should be situated. This track maintenance function makes it possible to determine a position of a predicted spot and a validation window around said predicted spot. Each spot appearing in the validation window of an existing TBD or transient track is a candidate for association with said track.
(73) In a step 401, the analysis module 102 initialises a variable l to 0. The variable makes it possible to enumerate the existing TBD or transient tracks.
(74) In a step 402, the analysis module 102 begins an analysis of the l.sup.th existing TBD or transient track, denoted track.sub.l. In the step 402, the analysis module 102 determines whether at least one TBD spot corresponds to the existing TBD or transient track track.sub.l. If such is the case, in one embodiment, the analysis module 102, in a step 403, allocates the TBD spot that is closest, according to the predefined metric, to the position of the predicted spot corresponding to the existing TBD or transient track track.sub.l.
(75) The step 403 is followed by a step 404 during which the analysis module 102 increments the variable l by one unit.
(76) If no TBD spot corresponds to the existing TBD or transient track track.sub.l, the step 402 is followed directly by the step 404.
(77) In a step 405, the analysis module 102 compares the value of the variable l with a variable L corresponding to the number of existing TBD or transient tracks. If the variable l equals the variable L, the analysis module 102 ends the step 40. Otherwise the analysis module 102 once again performs the step 402.
(78) In one embodiment, when an existing TBD or transient track is not updated during a predefined number of images, said track is abandoned by the analysis module 102.