Method of tracking a plurality of point-shaped objects in a three-dimensional space
10890521 · 2021-01-12
Assignee
Inventors
- Bernhard Wieneke (Goettingen, DE)
- Andreas Schröder (Goettingen, DE)
- Arne Henning (Goettingen, DE)
- Matteo Novara (Goettingen, DE)
- Daniel Schanz (Goettingen, DE)
Cpc classification
G06V10/7515
PHYSICS
G01P5/001
PHYSICS
G01N15/1456
PHYSICS
H04N23/74
ELECTRICITY
H04N23/90
ELECTRICITY
International classification
G01B11/00
PHYSICS
G01P5/00
PHYSICS
Abstract
For tracking a plurality of objects in a three-dimensional space two-dimensional pictures objects are recorded with two black and white cameras out of two different imaging directions. Both first pictures and second pictures of the two cameras are simultaneously exposed at two points in time in equal ways, a point in time at which the second pictures are exposed for a first time following to a point in time at which the first pictures are exposed for a last time at a much shorter interval than the two points in time of exposure of both the first and second pictures. First and second distributions of real positions of the individual objects are determined from their images in the first and second pictures, respectively; and temporally resolved trajectories of the individual objects in the three-dimensional space are determined from the first and second distributions of real positions.
Claims
1. A method of tracking a plurality of point-shaped objects in a three-dimensional space, the method comprising recording two-dimensional pictures of the plurality of objects by means of at least two black and white cameras out of at least two different imaging directions; wherein at least first and second pictures of the at least two black and white cameras are each exposed at at least two points in time in equal ways in that the objects are illuminated with short light pulses at the at least two points in time, wherein the first pictures are exposed simultaneously for a first time at a first point in time of a first light pulse of the short light pulses and for a last time at a second point in time of a second light pulse of the short light pulses following to the first point in time of the first light pulse of the short light pulses at a first inner-picture interval of time, wherein the second pictures are exposed simultaneously for a first time at a third point in time of a third light pulse of the short light pulses and for a last time at a fourth point in time of a fourth light pulse of the short light pulses following to the third point in time of the third light pulse of the short light pulses at a second inner-picture interval of time, and wherein the third point in time of the third light pulse of the short light pulses follows to the second point in time of the second light pulse of the short light pulses at an inner-picture interval of time of not more than 50% of the first inner-picture interval of time and of not more than 50% of the second inner-picture interval of time; determining a first distribution and a second distribution of real positions of the individual objects of the plurality of objects in the three-dimensional space from their images in the first pictures and from their images in the second pictures, respectively, wherein at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is determined from the images of the individual objects of the plurality of objects in the first or second pictures, respectively, in that an estimated distribution of the real positions of the individual objects in the three-dimensional space is adapted to their images in the first or second pictures, respectively; and determining temporally resolved trajectories of the individual objects of the plurality of objects in the three-dimensional space from the determined first and second distributions of the real positions of the individual objects, wherein different real positions of the same individual objects of the plurality of objects in the first distribution are interrelated, wherein different real positions of the same individual objects of the plurality of objects in the second distribution are interrelated, and wherein different real positions of the same individual objects of the plurality of objects in the first and second distributions are interrelated.
2. The method of claim 1, wherein the inter-picture interval of time at which the third point in time follows to the second point in time is not more than 20% of the first inner-picture interval of time and of not more than 20% of the second inner-picture interval of time.
3. The method of claim 2, wherein the inter-picture interval of time at which the third point in time follows to the second point in time is not more than 10% of the first inner-picture interval of time and of not more than 10% of the second inner-picture interval of time.
4. The method of claim 1, wherein the different real positions of the same individual objects of the plurality of objects in the first distribution and in the second distribution are interrelated between the first and second distributions, before the different real positions of the same individual objects of the plurality of objects in the first distribution are interrelated within the first distribution and the different real positions of the same individual objects of the plurality of objects in the second distribution are interrelated within the second distribution.
5. The method of claim 1, wherein third pictures of the at least two black and white cameras are each exposed at at least two points in time in equal ways, wherein the third pictures are exposed simultaneously for a first time at a fifth point in time and for a last time at a sixth point in time following to the fifth point in time at a third inner-picture interval of time, wherein the fifth point in time follows to the fourth point in time at the inter-picture interval of time of not more than 50% of the first inner-picture interval of time, of not more than 50% of the second inner-picture interval of time and of not more than 50% of the third inner-picture interval of time.
6. The method of claim 1, wherein at least one of velocities of the individual objects of the plurality of objects, and accelerations of the individual objects of the plurality of objects are determined from the temporally resolved trajectories of the individual objects of the plurality of objects.
7. The method of claim 1, wherein at least one of a velocity field of a flow with which the individual objects of the plurality of objects move, an acceleration field of a flow with which the individual objects of the plurality of objects move, and a pressure field of a flow with which the individual objects of the plurality of objects move are determined from the temporally resolved trajectories of the individual objects of the plurality of objects.
8. The method of claim 1, wherein the pictures of the black and white cameras are exposed at the at least two points in time in that the objects of the plurality of objects are illuminated by light pulses.
9. The method of claim 1, wherein the estimated distribution of the real positions of the individual objects in the three-dimensional space is an object distribution of all of the individual objects in the three-dimensional space for which trajectories in the three-dimensional space have been determined before.
10. The method of claim 1, wherein the estimated distribution of the real positions of the individual objects in the three-dimensional space is estimated by extrapolating a starting distribution of the individual objects in the three-dimensional space along the trajectories of the individual objects in the three-dimensional space for the at least two points in time at which the respective first or second pictures are exposed.
11. The method of claim 10, wherein the starting distribution is a distribution which has been determined from images of the individual objects of the plurality of objects in other pictures of the black and white cameras which have been exposed at least two other points in time, or a momentary distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space which has been determined from images of the individual objects of the plurality of objects in other pictures of the black and white cameras which have been exposed at one other point in time.
12. The method of claim 1, wherein an auto-correlation function of at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is calculated to interrelate the different real positions of the same individual objects of the plurality of objects in the first or the second distribution, respectively.
13. The method of claim 1, wherein a cross-correlation function of the first and the second distributions of the real positions of the individual objects of the plurality of objects in the three-dimensional space is calculated to interrelate the different real positions of the same individual objects of the plurality of objects in the first and second distributions.
14. A method of tracking a plurality of point-shaped objects in a three-dimensional space, the method comprising recording two-dimensional pictures of the plurality of objects by means of at least two black and white cameras out of at least two different imaging directions; wherein at least first and second pictures of the at least two black and white cameras are each exposed at at least two points in time in equal ways in that the objects are illuminated with short light pulses at the at least two points in time, wherein the first pictures are exposed simultaneously for a first time at a first point in time of a first light pulse of the short light pulses and for a last time at a second point in time of a second light pulse of the short light pulses following to the first point in time of the first light pulse of the short light pulses at a first inner-picture interval of time, wherein the second pictures are exposed simultaneously for a first time at a third point in time of a third light pulse of the short light pulses and for a last time at a fourth point in time of a fourth light pulse of the short light pulses following to the third point in time of the third light pulse of the short light pulses at a second inner-picture interval of time, and wherein the third point in time of the third light pulse of the short light pulses follows to the second point in time of the second light pulse of the short light pulses at an inner-picture interval of time of not more than 50% of the first inner-picture interval of time and of not more than 50% of the second inner-picture interval of time; determining a first distribution and a second distribution of real positions of the individual objects of the plurality of objects in the three-dimensional space from their images in the first pictures and from their images in the second pictures, respectively, wherein at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is determined from the images of the individual objects of the plurality of objects in the first or second pictures, respectively, in that an estimated distribution of the real positions of the individual objects in the three-dimensional space is adapted to their images in the first or second pictures, respectively, and wherein the estimated distribution of the real positions of the individual objects in the three-dimensional space is an object distribution of all of the individual objects in the three-dimensional space for which trajectories in the three-dimensional space have been determined before; and determining temporally resolved trajectories of the individual objects of the plurality of objects in the three-dimensional space from the determined first and second distributions of the real positions of the individual objects, wherein different real positions of the same individual objects of the plurality of objects in the first distribution are interrelated, wherein different real positions of the same individual objects of the plurality of objects in the second distribution are interrelated, and wherein different real positions of the same individual objects of the plurality of objects in the first and second distributions are interrelated.
15. A method of tracking a plurality of point-shaped objects in a three-dimensional space, the method comprising recording two-dimensional pictures of the plurality of objects by means of at least two black and white cameras out of at least two different imaging directions; wherein at least first and second pictures of the at least two black and white cameras are each exposed at at least two points in time in equal ways in that the objects are illuminated with short light pulses at the at least two points in time, wherein the first pictures are exposed simultaneously for a first time at a first point in time of a first light pulse of the short light pulses and for a last time at a second point in time of a second light pulse of the short light pulses following to the first point in time of the first light pulse of the short light pulses at a first inner-picture interval of time, wherein the second pictures are exposed simultaneously for a first time at a third point in time of a third light pulse of the short light pulses and for a last time at a fourth point in time of a fourth light pulse of the short light pulses following to the third point in time of the third light pulse of the short light pulses at a second inner-picture interval of time, and wherein the third point in time of the third light pulse of the short light pulses follows to the second point in time of the second light pulse of the short light pulses at an inner-picture interval of time of not more than 20% of the first inner-picture interval of time and of not more than 20% of the second inner-picture interval of time; determining a first distribution and a second distribution of real positions of the individual objects of the plurality of objects in the three-dimensional space from their images in the first pictures and from their images in the second pictures, respectively; and determining temporally resolved trajectories of the individual objects of the plurality of objects in the three-dimensional space from the determined first and second distributions of the real positions of the individual objects, wherein different real positions of the same individual objects of the plurality of objects in the first distribution are interrelated, wherein different real positions of the same individual objects of the plurality of objects in the second distribution are interrelated, and wherein different real positions of the same individual objects of the plurality of objects in the first and second distributions are interrelated.
16. The method of claim 15, wherein at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is determined by triangulation from the images of the individual objects of the plurality of objects in the first or second pictures, respectively.
17. The method of claim 15, wherein at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is tomographically determined from the images of the individual objects of the plurality of objects in the first or second pictures, respectively.
18. The method of claim 15, wherein at least the first or the second distribution of the real positions of the individual objects of the plurality of objects in the three-dimensional space is determined from the images of the individual objects of the plurality of objects in the first or second pictures, respectively, in that an estimated distribution of the real positions of the individual objects in the three-dimensional space is adapted to their images in the first or second pictures, respectively.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The invention can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention. In the drawings, like reference numerals designate corresponding parts throughout the several views.
(2)
(3)
(4)
(5)
DETAILED DESCRIPTION
(6) In a method according to the invention of tracking a plurality of point-shaped objects, two-dimensional pictures of the plurality of the objects are taken or recorded with several cameras out of several imaging directions. The cameras are black and white or monochrome cameras. Even if in the following cameras are mentioned without indicating that they are black and white or monochrome cameras, only such black and white cameras are meant as long as nothing else is explicitly indicated. The term black and white cameras or monochrome cameras does not mean that no color or wavelength filter may be arranged in front of the cameras. Instead such a filter may be used to, for example, suppress background light over monochromatic light by which the objects are purposefully illuminated. Even in this case, however, the several cameras do not serve for spectrally resolving the light from the plurality of objects.
(7) Typically, the several cameras by which the pictures of the objects are recorded out of different imaging directions are arranged at different locations, i.e. separated spatially. In principle, however, the method according to the invention may also be carried out using a so-called Lightfield camera. A Lightfield camera comprises an array of objective lenses for imaging or mapping a measurement volume out of slightly different imaging directions onto different areas of a picture sensor. Each of these areas of the picture sensor and the associated objective lens of the array of objective lenses is regarded as being a camera, here.
(8) The cameras are synchronized with regard to the succession of their pictures at least to such an extent that the pictures of the different cameras are always exposed simultaneously. At least two consecutive pictures of each of the cameras are exposed at intervals in time. Particularly, each of the at least two consecutive pictures of each camera is at least exposed twice at an inner-picture interval of time, and it may also be exposed at at least one further point in time, i.e. at least trice at three points in time at inner-picture intervals of time.
(9) Even for tracking objects which move very fast in a three-dimensional space, the method according to the invention may be carried out with low cost or value for money cameras which have a comparatively low picture frequency. In principle, the method according to the invention could even be carried out with cameras which only allows for recording two consecutive pictures. In any case, it is favorable if the cameras only have short dead times during picture changes between their at least two pictures, in which they do not register incident light.
(10) In the method according to the invention, real positions of the individual objects in the three-dimensional space are determined from their image in the respective simultaneously exposed pictures. The real positions are determined for essentially all images of the objects in the pictures recorded out of the different imaging directions. As each of the at least two pictures of the cameras has been exposed at least twice, a distribution of real positions of the individual objects in the three-dimensional space is determined from the images of the objects in the pictures of the cameras, which includes more real positions than individual objects present. Particularly, the number of the real positions is at least about two times the number of the individual objects present. The real positions belonging to one of the objects are then interrelated to determine a temporally resolved trajectory of the respective object in the three-dimensional space. This step of determining includes interrelating different positions of each object within each of the distributions.
(11) Surprisingly, it has been found that not only double exposed pictures of essentially two-dimensional light sections in which the images of objects directly correspond to a position of the objects in the respective light section may be evaluated but that this is also possible for multiple exposed pictures of objects in a volume which have been recorded out of different imaging directions. By means of the multiple exposure of the pictures of the cameras, objects may be tracked which quickly alter their positions because they, for example, move with a fast flow, even when using cameras with standard picture frequency. The extension of the sequence of the exposures beyond one picture change of the cameras provides an additional temporal information as the images of the objects in the previously recorded pictures correspond to prior real positions of the objects along their trajectories than the images in the pictures recorded later.
(12) It is to be understood that the objects may only have such a high density in the respective volume and their images may only have such a high density in the pictures recorded that real positions of the objects in the three-dimensional space can be determined from the images and that these real positions in the three-dimensional space can be interrelated. By using suitable methods for determining the real positions of the objects in the in the three-dimensional space and for interrelating them to determine the trajectories of the objects, as they are indicated in the following, however, sufficiently high densities of the objects in the volume are possible to, for example, spatially resolve even complex flows with regard to their velocity, acceleration and pressure fields. Particularly, particle densities of more than 0.05 ppp (particles per pixel of the picture sensors of the cameras) are possible.
(13) In the method according to the invention, real positions of the objects in the space are determined for essentially all images of the objects in the simultaneously multiple exposed pictures, and only these real positions in the space are interrelated to determine the trajectories each belonging to one of the objects. In other words, the images of the objects are not yet interrelated, neither within the multiple exposed pictures nor between the pictures. This does, however, not exclude that information acquired from interrelating the real positions to determine the trajectories of the individual objects is fed back to the step of determining the real positions of the individual objects from their images in the pictures. Instead such information may be used to determine these real positions at a higher precision. It is also possible to then feed back these real positions of higher precision to the step of interrelating, particularly to increase the security in associating different real positions with one object.
(14) In the method according to the invention, the pictures of the cameras are exposed at the at least two points in time at an interval in time in a same way so that the images of one and the same real object in the individual pictures may not be differentiated. Thus, it is not possible to conclude on the points in time of exposure of the cameras which resulted in the different images of the individual objects. Instead, in the method according to the invention, differentiating the images with regard to different times and objects generally occurs by interrelating the real positions each belonging to one of the objects to a trajectory of the respective object in the three-dimensional space. This step of interrelating does not require that the real positions belonging to the respective one of the objects can be differentiated from the real positions of the other objects in the three-dimensional space prior to that step. Instead, the step of interrelating of the method according to the invention includes the analysis which of the real positions are associated with which of the individual objects.
(15) In the method according to the invention, third and also further pictures of the at least two black and white cameras may each exposed at at least two points in time in equal ways. The third pictures are exposed simultaneously for a first time at a fifth point in time and for a last time at a sixth point in time following to the fifth point in time at a third inner-picture interval of time, wherein the fifth point in time follows to the fourth point in time at the inter-picture interval of time of not more than 50% of the first inner-picture interval of time, of not more than 50% of the second inner-picture interval of time and of not more than 50% of the third inner-picture interval of time. With the third or even further pictures it becomes clear that the points in time at which the pictures are exposed at least simultaneously are grouped around the picture changes.
(16) In the method according to the invention, an additional picture of each of the cameras may only be exposed once. Both of the at least two consecutive pictures of each of the camera, however, are exposed at the at least two points in time at the inner-picture interval in time.
(17) In the method according to the invention, each of the at least two consecutive pictures of each camera is exposed at at least two points in time at an inner-picture interval in time, which is at least two times, five times or even ten time an inter-picture interval at which the first point in time of exposing the later picture follows to the last point in time of exposing the prior of the at least consecutive pictures. In other words, the first point in time of exposing the later picture follows to the last point in time of exposing the prior picture at the inter-picture interval of time of not more than 50% of the inner-picture intervals of times of both the prior and the later picture. Thus, with a sequence of multiple exposed pictures recorded with the cameras, the at least two points in time at which the pictures are exposed are close in time to the picture changes between the consecutive pictures and thus to the associated short dead times of the cameras.
(18) The short inter-picture intervals have the effect that the objects may not have moved far between the associated points in time of exposing the pictures of the cameras so that a simple interrelation of the associates real positions in the three-dimensional space is possible. The real positions at the respective points in time arranged at a longer interval in time may not be as easily interrelated to determine the trajectory of the respective object. The longer distances of these real positions of the objects, however, allow for a more precise determination of the velocities of the objects and, with determining several velocities from several real positions along each trajectory, also of the accelerations of the objects.
(19) As already mentioned above, the method according to the invention is particularly suitable for determining velocities and accelerations of the individual objects and for determining velocity fields, acceleration fields and pressure fields of a flow with which the objects move from the trajectories of the objects. For the latter determinations, the objects may be seeded into the flow at a suitable seeding density. If the objects then have a mass density similar to the fluid density they will closely follow the flow when the pictures of the particles are recorded. The objects may be objects with which flows are usually seeded to measure them optically. Both in flowing gases and in flowing liquids, these objects include solid particles, fine droplets of liquids and fine gas bubbles.
(20) The pictures of the cameras may be exposed by opening a shutter of the respective camera. In principle, the shutter may be a mechanical shutter. Electronic shutters may be controlled and opened faster and particularly at a higher frequency. Very quickly consecutive points in time of exposing the pictures of the cameras, however, are preferably not realized by a shutter but in that the objects are illuminated with short light pulses at the respective points in time, the intensity of the light pulses being sufficient for imaging the objects within the short times of illumination. For example, laser light pulses provided by lasers with a sufficiently high output power are suitable for this purpose. Here, it proves to be advantageous that many commercially available lasers may be operated at least two times at a short interval in time for outputting a laser light pulse of high light power. Instead of lasers, however, LEDs and LED arrays may also be used as light sources for providing suitable light pulses. Other light sources, like for example laser diodes, may also be used.
(21) From the images of the individual objects in the pictures of the cameras which have been exposed simultaneously at the at least two points in time, the distribution of the real positions of the individual objects in the three-dimensional space may be determined by triangulation. The distribution of the positions of the individual objects in the three-dimensional space will then be the basis for interrelating the real positions to determine trajectories of the individual objects.
(22) Alternatively, the distribution of the real positions of the individual objects in the three-dimensional space may be determined tomographically from the pictures of the cameras which have been exposed simultaneously at the at least two points in time, like it is, for example, known from US patent application publication US 2005/0062954 A1 and the corresponding U.S. Pat. No. 7,382,900.
(23) The result of such a tomographic analysis of the simultaneously exposed pictures may more precisely be designated as a density distribution of the objects over the voxels of a measurement volume. If in the present description real positions of the objects in the three-dimensional space are mentioned, this shall, however, always include the possibility that these real positions in the three-dimensional space are described as a density distribution of the objects over voxels of a measurement volume.
(24) A further, advantageous procedure of determining the distribution of the real positions of the individual objects in the three-dimensional space includes adapting an estimated distribution of the real positions of the individual objects in the three-dimensional space to their images in the pictures of the cameras which have been exposed simultaneously. This procedure is, for example, known from DE 10 2009 009 551 B4 (corresponding to U.S. Pat. No. 8,542,882 B2) or WO 2014/191461 A1 (corresponding to U.S. Pat. No. 9,733,267 B2). It is, however, to be understood that the estimated distribution has to include one position for each of the individual objects and for each time of exposure of the respective picture. This means, with two illuminations of the simultaneously exposed pictures, the distribution has to have twice as many real positions than individual objects, as each individual object is imaged at two points in time and thus at two positions if it has moved between the consecutive exposures of the pictures.
(25) In so far as the teaching of WO 2014/191461 A1 (corresponding to U.S. Pat. No. 9,733,267 B2) is applied in carrying out the present invention, the estimated distribution for a set of simultaneously exposed pictures of the cameras may be estimated from a distribution which has been determined from the images of the individual objects in other pictures of the cameras which have been exposed at at least two other points in time, or from a momentary distribution determined for one point in time at which further pictures have been exposed only once. This step of estimating may include a step of extrapolating the real positions of the individual objects in the three-dimensional space along the trajectories of the individual objects in the three-dimensional space for the at least two points in time. This also provides for a basis for the step of continuously interrelating the real positions to determine the trajectories of the individual objects in the space.
(26) The step of interrelating the real positions of the individual objects to determine their trajectories may further or instead also be based on a calculation of an auto-correlation function of each of the distributions of the real positions of the individual objects in the three-dimensional space which has been determined from the images of the objects in the pictures multiple exposed at the at least two points in time. Correspondingly, a cross-correlation function of two distributions of the real positions of the individual objects in the three-dimensional space, which have been determined from the images of the objects in the at least two consecutive pictures, may be calculated to interrelate the different positions of each respective object at the points in time of exposure of the at least two consecutive pictures of each camera.
(27) Now referring in greater detail to the drawings, a measurement arrangement 1 depicted in
(28) The pictures of the cameras 2 and 3 are exposed by illuminating the objects 9 in the measurement volume 6 by means of a light source 11. In the present embodiment, the light source 11, besides two pulsed lasers 12 and a projection lens 14, includes two light guides 13. Instead of the two light guides 13, a polarization beam splitter may be used for merging the laser light from the two lasers 12 in front of the projection lens 14. In this case, the laser light from the two lasers 12 will be polarized differently. The lasers 12 are alternately controlled or operated such that the objects 9 are illuminated by light pulses. The pictures of the cameras 2 and 3 are exposed by that light of the light pulses that is reflected by the objects 9. As the measurement volume 6 is otherwise not illuminated, the pictures of the cameras 2 and 3 are only exposed at the points in time of the light pulses from the lasers 12. In principle, one laser 12 would be sufficient for the light source 11. Due to the second laser 12, however, it is easier to illuminate the measurement volume 6 twice in quick succession each time with a light pulse of high intensity from one of the two lasers 12.
(29) From the images of the objects 9 in the pictures recorded by the picture sensors 5 which show the measurement volume 6 out of the different imaging directions 7, the real positions of the objects 9 in the measurement volume 6 are determined. As already explained above, this determination may be carried out in different ways.
(30)
(31)
(32) Even if the light integration periods of time 15 and 16 in
(33) The flow chart according to
(34) The flow chart according to
(35) In the method according to the invention, no positions of images of the objects 9 in the pictures recorded with picture sensors 5 are interrelated. Instead the real positions of the objects 9 in the three-dimensional space determined from these images are interrelated. This has to be considered when looking at
(36) Many variations and modifications may be made to the preferred embodiments of the invention without departing substantially from the spirit and principles of the invention. All such modifications and variations are intended to be included herein within the scope of the present invention, as defined by the following claims.