Method for corrected depth measurement with a time-of-flight camera using amplitude-modulated continuous light

Abstract

A method for corrected depth measurement with a time-of-flight camera using amplitude-modulated continuous light. In order to enable an accurate and efficient depth measurement with a time-of-flight camera, the method includes, for each of a plurality of pixels of a sensor array of the camera: acquiring with the camera a raw depth value r.sub.m for the pixel; and automatically calculating a ground truth value r.sub.t according to: r.sub.t=g(r.sub.m−c.sub.m)+c.sub.t, to correct a systematic depth error of the raw depth value r.sub.m, wherein c.sub.m is a pixel-dependent first offset, g is a pixel-independent first function and c.sub.t is a pixel-independent second offset.

Claims

1. A method for corrected depth measurement, the method comprising: acquiring, with a time-of-flight camera using amplitude-modulated continuous light, a raw depth value r.sub.m for each of a plurality of pixels of a sensor array of the camera; and automatically calculating for each of the plurality of pixels a ground truth value r.sub.t according to:
r.sub.t=g(r.sub.m−c.sub.m)+c.sub.t, to correct a systematic depth error of the raw depth value r.sub.m, wherein c.sub.m is a pixel-dependent first offset, g is a pixel-independent first function and c.sub.t is a pixel-independent second offset.

2. A method according to claim 1, wherein acquiring the raw depth value r.sub.m comprises determining four amplitude samples at a sampling frequency four times higher than a modulation frequency of the amplitude-modulated continuous light.

3. A method according to claim 1, wherein the first function g is applied by accessing a look-up table representing the first function g.

4. A method according to claim 3, wherein the look-up table is calculated by applying a second function ƒ, which is the inverse function of the first function g.

5. A method according to claim 4, wherein the second function ƒ comprises a linear term, a third order harmonic term and a fifth order harmonic term with respect to the modulation frequency.

6. A method according to claim 1, wherein before the corrected depth measurement, a calibration is performed for the camera, in which at least one of the first offset c.sub.m, the first function g and the second offset c.sub.t is determined.

7. A method according to claim 6, wherein in the calibration comprises a general calibration, in which at least one of the first offset c.sub.m and the first function g is determined only once for a plurality of cameras.

8. A method according to claim 6, wherein the first function g is applied by accessing a look-up table representing the first function g, the look-up table is calculated by applying a second function ƒ, which is the inverse function of the first function g, and the calibration comprises determining the second function ƒ by: performing with the camera a plurality of different depth measurements, each depth measurement providing a raw depth value r.sub.m(k) for each of a plurality of pixels in an area of interest, wherein k=1, . . . , N is the number of the depth measurement; for each depth measurement and for each pixel in the area of interest, determining a ground truth value r.sub.t (k); defining the second offset c.sub.t; for each pixel in the area of interest, defining a pixel-dependent third function ƒ.sub.p with at least one parameter and fitting the at least one parameter to the condition r.sub.m(k)=c.sub.m+ƒ.sub.p (r.sub.t(k)−c.sub.t); and determining the second function ƒ based on the third functions ƒ.sub.p of a plurality of pixels in the area of interest.

9. A method according to claim 8, wherein the area of interest corresponds to a portion of the sensor array.

10. A method according to claim 8, wherein the second function ƒ is determined by averaging the at least one parameter of the third functions ƒ.sub.p over a plurality of pixels in the area of interest.

11. A method according to claim 8, wherein the calibration comprises, for each pixel in the area of interest and each depth measurement, calculating an offset estimate c.sub.m(k) for the first offset c.sub.m according to c.sub.m(k)=r.sub.m−ƒ(r.sub.t(k)−c.sub.t).

12. A method according to claim 1, wherein the first offset c.sub.m is determined by averaging the offset estimate c.sub.m(k) over a plurality of depth measurements.

13. A method according to claim 6, wherein the calibration comprises using the first function g and the first offset c.sub.m determined in a general calibration with one camera and performing an individual calibration for a different camera by: performing a depth measurement for at least one pixel to obtain a raw depth value r.sub.m; determining a ground truth value r.sub.t for the at least one pixel; and calculating the second offset c.sub.t according to c.sub.t=r.sub.t−g(r.sub.m−c.sub.m).

14. A method according to claim 1, wherein the second offset c.sub.t is temperature-dependent.

15. A method according to claim 6, wherein the calibration comprises determining the second offset c.sub.t, for a first temperature and, for each of at least one second temperature: performing a depth measurement for at least one pixel to obtain a raw depth value r.sub.m; determining a ground truth value r.sub.t for the at least one pixel; and calculating the second offset c.sub.t for the respective second temperature according to c.sub.t=r.sub.t−g(r.sub.m−c.sub.m); and determining at least one parameter related to a temperature dependency of the second offset c.sub.t.

16. A system for corrected depth measurement with a time-of-flight camera using amplitude-modulated continuous light, the system comprising: a sensor array having a plurality of pixels; processing circuitry, coupled to the sensor array, the processing circuitry being configured to carry out the steps of: acquiring a raw depth value r.sub.m for each of the plurality of pixels of the sensor array; and automatically calculating for each of the plurality of pixels a ground truth value r.sub.t according to:
r.sub.t=g(r.sub.m−c.sub.m)+c.sub.t, to correct a systematic depth error of the raw depth value r.sub.m, wherein c.sub.m is a pixel-dependent first offset, g is a pixel-independent first function and c.sub.t is a pixel-independent second offset.

17. A system according to claim 16, wherein acquiring the raw depth value r.sub.m comprises determining four amplitude samples at a sampling frequency four times higher than a modulation frequency of the amplitude-modulated continuous light.

18. A system according to claim 16, wherein before the corrected depth measurement, a calibration is performed for the camera, in which at least one of the first offset c.sub.m, the first function g and the second offset c.sub.t is determined.

19. A system according to claim 18, wherein the first function g is applied by accessing a look-up table representing the first function g, the look-up table is calculated by applying a second function ƒ, which is the inverse function of the first function g, and the calibration comprises determining the second function ƒ by: performing with the camera a plurality of different depth measurements, each depth measurement providing a raw depth value r.sub.m(k) for each of a plurality of pixels in an area of interest, wherein k=1, . . . , N is the number of the depth measurement; for each depth measurement and for each pixel in the area of interest, determining a ground truth value r.sub.t(k); defining the second offset c.sub.t; for each pixel in the area of interest, defining a pixel-dependent third function ƒ.sub.p with at least one parameter and fitting the at least one parameter to the condition r.sub.m(k)=c.sub.m+ƒ.sub.p(r.sub.t(k)−c.sub.t); and determining the second function ƒ based on the third functions ƒ.sub.p of a plurality of pixels in the area of interest.

20. A non-transitory computer program product stored on a computer-readable medium and comprising computer-implementable instructions which, when executed by a processor, cause the processor to carry out the method as claimed in claim 1.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Further details and advantages of the present invention will be apparent from the following detailed description of not limiting embodiments with reference to the attached drawing, wherein:

(2) FIG. 1 is a schematic view of a TOF camera that can be used for the inventive method; and

(3) FIG. 2 is a flowchart illustrating an embodiment of the inventive method.

DETAILED DESCRIPTION

(4) FIG. 1 schematically shows a ToF camera 1 that is adapted for depth measurement using amplitude-modulated continuous light. It comprises a rectangular sensor array 2 with a plurality (e.g. several thousand or several ten thousand) of pixels 3. Furthermore, it comprises a memory 5 and a processing unit 6. The camera 1 is configured to emit amplitude-modulated continuous light using one or several light emitters, which are not shown here for sake of simplicity. The light is reflected by a 3D object or scenery in a field of view of the camera 1 and the reflected light is received by the pixels 3 of the sensor array 2. The amplitude of the received light is sampled at a frequency four times higher than a modulation frequency of the light. In other words, four amplitudes A.sub.0 . . . 3, also referred to as taps, are used to retrieve the phase φ of the modulated light, since

(5) $\begin{array}{cc}\phi =\mathrm{atan}\left(\frac{A_0-A_2}{A_1-A_3}\right).& \left(\mathrm{eq}.2\right)\end{array}$

(6) Since the sampling frequency is four times higher than the fundamental modulation frequency, according to the Shannon-Nyquist theorem, aliasing can occur for harmonics with an order greater than two. When a certain raw depth value r.sub.m is measured for one of the pixels 3, this aliasing, along with other effects, generally leads to a deviation from a ground truth value r.sub.t. This deviation is corrected according to an inventive method which will now be described with reference to the flowchart in FIG. 2.

(7) According to the inventive method, the relation between the raw depth value r.sub.m and the ground truth value r.sub.t is given by
r.sub.m=c.sub.m+ƒ(r.sub.t−c.sub.t),  (eq. 3)
in this case, the second function ƒ is modeled as:
ƒ(r)=a.sub.0+r+b.sub.1 cos(k.sub.4r+θ.sub.1)+b.sub.2 cos(k.sub.8r+θ.sub.2)+ . . .   (eq. 4)
(eq. 3) can be rewritten as follows:
r.sub.t=g(r.sub.m−c.sub.m)+c.sub.t  (eq. 1)
so that the ground truth value r.sub.t can be calculated from the raw depth value r.sub.m. However, before this correction can be applied in a corrected depth measurement 500, the first function g as well as the first and second offset c.sub.m, c.sub.t need to be determined in a calibration 100. The calibration 100 comprises a general calibration 200 that needs to be carried out for only one camera 1 of a given production series, which may be referred to as a “golden sample”. This general calibration 200 yields a first function g and a first offset c.sub.m that can be used for all cameras 1 of this production series. Furthermore, it yields a second offset c.sub.t with a sample-independent part c.sub.t1(T) that can be used for all cameras 1 of the production series.

(8) In a first step, at 210, an area of interest 4 comprising a plurality of pixels 3 is defined on the sensor array 2. In this example, the area of interest 4 is rectangular and centred with respect to the sensor array 2, but it could have a different shape and/or location. The area of interest 4 represents only a small portion of the sensor array 2 and comprises e.g. between 10 and 20 pixels. Also, the second offset c.sub.t is defined, i.e. it is chosen arbitrarily.

(9) In another step, at 220, a plurality of different depth measurements are performed. For example, the camera 1 could be positioned opposite a flat surface and the distance could be increased between consecutive depth measurements. If a temperature dependency of the second offset c.sub.t is taken into account—which is normally the case—, the temperature has to be maintained constant until the depth measurements have been performed. Also, the temperature has to be determined, e.g. measured. In each depth measurement, a raw depth value r.sub.m(k) and a ground truth value r.sub.t(k) are determined for each pixel 3 in the area of interest 4. The raw depth value r.sub.m(k) is determined by the camera 1, while the ground truth value r.sub.t(k) is measured and/or calculated independently of the camera 1. All quantities mentioned here and in the following can be stored in the memory 5 of the camera 1 while all necessary calculations can be performed by the processing unit 6.

(10) In a next step, at 230, several pixel-dependent third functions ƒ.sub.p are defined, which have a similar form as the first function ƒ and comprise corresponding parameters. Using the previously determined raw depth values r.sub.m(k) and ground truth values r.sub.t(k) for the respective pixel 3, the parameters of the individual third function ƒ.sub.p are fitted to fulfill the condition
r.sub.m(k)=c.sub.m+ƒ.sub.p(r.sub.t(k)−c.sub.t)  (eq. 5)
for all depth measurements. This yields a plurality of third functions ƒ.sub.p, namely one for each pixel 3 in the area of interest 4, which are generally pairwise different.

(11) At 240, the parameters of the individual third functions ƒ.sub.p are averaged over all pixels 3 in the area of interest 4 in order to obtain the parameters for the second function ƒ, which is now pixel-independent. As an alternative to defining and determining individual third functions ƒ.sub.p as in step 230, the second function ƒ could be directly determined based on the raw depth values r.sub.m(k) and ground truth values r.sub.t(k) of all pixels (3). In this case, step 240, would be obsolete.

(12) In another step at 250, an offset estimate c.sub.m(k) is calculated for every pixel 3 of the sensor array 2 and for each depth measurement by
c.sub.m(k)=r.sub.m(k)−ƒ(r.sub.t(k)−c.sub.t)  (eq. 6)
and at 260, the average of all offset estimates is taken over the depth measurements to obtain the first offset c.sub.m for the individual pixel 3.

(13) With the second function ƒ known, its inverse function, namely the first function g, can be determined. This is done at 270 by calculating a look-up table for the first function g. Therefore, a set of theoretical values r′.sub.t(k)=ku, k=0, . . . , N−1 for the ground truth depth is defined with a chosen resolution u in the unambiguity range of the camera 1, e.g. u=1 mm, k=0, . . . , 7500. If the function values of r′.sub.t(k) for the second function ƒ are also defined with respect to the resolution u, i.e. r′.sub.m=ƒ(r′.sub.t)=ƒ(ku)=lu, the inverse function r′.sub.t=g(r′.sub.m) can be expressed as r′.sub.t=ku=g(r′.sub.m)=g (lu)=LUT (l)u. The look-up table is determined in a simple iterative algorithm.

(14) If the temperature dependency of the second offset c.sub.t can be neglected, the general calibration 200 ends with this step and the method continues at 360, as indicated by the dash-dotted arrow. Normally though, the second offset c.sub.t has to be treated as temperature-dependent, and the general calibration 200 continues with at least one second temperature that is different from the temperature of the previous measurements. The temperature dependency of the second offset c.sub.t can be represented by a function having several parameters. For example, the temperature dependency can be assumed to be linear, wherefore the respective function has two parameters. In order to determine the parameters, one needs at least the same number of values for the second offset c.sub.t, one of which has already been defined previously. In general, the temperature dependency of the second offset may be represented by:
c.sub.t=c.sub.t(T.sub.0)+b(T−T.sub.0)+c(T−T.sub.0).sup.2+ . . . =c.sub.t0+c.sub.t1(T)  (eq. 8)
where c.sub.t0=c.sub.t(T.sub.0) is a temperature-independent constant and c.sub.t1(T) represents all temperature-dependent terms, i.e. linear and possible higher order terms. At 290, the temperature is changed. Another depth measurement is performed to obtain a raw depth value r.sub.m for at least one pixel 3, preferably a plurality of pixels 3, at 300. For instance, this could be all pixels 3 in the area of interest 4. Likewise, at 310, a ground truth value r.sub.t is determined for each pixel 3. At 320, the second offset c.sub.t is calculated for every pixel 3. If more than one pixel 3 has been taken into account, the average over all pixels 3 is taken at 330 to determine the final value for the second offset c.sub.t. If it is decided at 340 that the second offset c.sub.t has to be determined for another second temperature, the temperature is changed again at 290 and the following steps are repeated. If the measurement has been performed for all second temperatures, at least one parameter defining the temperature dependency of the second offset c.sub.t is determined at 350, and the general calibration 200 ends.

(15) As mentioned above, the results of the general calibration 200 can not only be used for a single camera 1, but for all cameras 1 of the same production series. If a different camera 1 (i.e. different from the camera 1 used in the general calibration 200) needs to be calibrated, which is checked at 360, the method continues with an individual calibration 400. In this individual calibration 400, another depth measurement is performed to obtain a raw depth value r.sub.m for at least one pixel 3, preferably a plurality of pixels 3, at 410. For instance, this could be all pixels 3 in the area of interest 4. Likewise, at 420, a ground truth value r.sub.t is determined for each pixel 3. At 430, the second offset c.sub.t is calculated for every pixel 3. If more than one pixel 3 has been taken into account, the average over all pixels 3 is taken at 440 to determine the final value for the second offset c.sub.t.

(16) If, as is normally the case, the second offset c.sub.t is represented not only by a temperature-independent constant c.sub.t0 but also by a temperature-dependent part c.sub.t1(T), it can be assumed that the temperature dependency for this camera differs from the “golden sample” only by a constant offset, i.e. the temperature-independent constant c.sub.t0 is device-dependent and generally different for each camera, but the temperature-dependent part c.sub.t1(T) is device-independent. Therefore, it is sufficient to compare the value for the second offset c.sub.t determined for the temperature of the individual calibration 400 with the value that would have been valid for the “golden sample” and shift the entire function by the difference (if present). In other words, only the temperature-independent constant c.sub.t0 of the temperature-dependent function has to be determined or “updated”, while the temperature-dependent part c.sub.t1(T), e.g. linear or quadratic terms, can be left unchanged. Therefore, even if the second offset c.sub.t is temperature-dependent, the individual calibration 400 can be carried out with a single depth measurement. Since the temperature does not change (in any significant way) during a single depth measurement, no problems arise with maintaining a constant temperature during the individual calibration 400.

(17) When the calibration 100 has been completed, the actual corrected depth measurement 500 can begin. At 510, raw depth values r.sub.m are acquired for all pixels 3 of the sensor array 2 and at 520, the corresponding ground truth values r.sub.t are calculated according to eq. 1. This can be repeated if a new corrected depth measurement is required at 530. If not, the method ends.