Time-of-flight camera

Abstract

The disclosure relates to a time-of-flight camera comprising: a time-of-flight sensor having several time-of-flight pixels for determining a phase shift of emitted and captured light, distance values being determined in accordance with the detected phase shifts, characterised in that the time-of-flight camera has a memory in which parameters of a point spread function, which characterise the time-of-flight camera and the time-of-flight sensor, are stored; an evaluation unit which is designed to deploy a detected complex-valued image in Fourier space, in accordance with the stored point-spread function, and a complex-valued image corrected by diffused light is determined and the phase shifts or distance values are determined using the corrected complex-valued image.

Claims

1. A time-of-flight camera for a time-of-flight camera system, comprising: a time-of-flight sensor with a plurality of time-of-flight pixels for determining a phase shift of emitted and received light, wherein based on the detected phase shifts distance values are determined; a memory in which at least parameters of a point spread function are stored, wherein the point spread function considers a scattered light behavior and signal crosstalk of the time-of-flight camera and the time-of-flight sensor; and an evaluation unit, which is configured such that based on the stored point spread function a detected image is convoluted and a corrected image is determined, wherein the phase shifts or distance values are determined from the corrected image, wherein the point spread function stored in the memory has been determined in which a point light source and the time-of-flight camera are disposed such that the time-of-flight sensor of the time-of-flight camera detects the point light source, wherein a distance between the point light source and the time-of-flight camera and/or a beam profile of the point light source are selected such that on the time-of-flight sensor less than 5 time-of-flight pixels in a pixel row or column or a maximum of 16×16 pixels are illuminated, and wherein the point spread function is determined at least based on a subset of the time-of-flight pixels of the time-of-flight sensor.

2. The time-of-flight camera according to claim 1, wherein the point spread function is complex-valued.

3. The time-of-flight camera according to claim 1, wherein the deconvolution of the detected image and the stored point spread function is implemented in the Fourier space.

4. The time-of-flight camera according to claim 1, wherein the detected image is reduced in resolution and a correction is determined with this reduced resolution, whereupon the correction is scaled up to the original resolution of the detected image and the detected image is corrected with the scaled-up correction.

5. The time-of-flight camera according to claim 4, wherein the reduction of the resolution is carried out by averaging amplitudes of adjacent pixels and the up-scaling is implemented by duplication of the amplitudes.

6. The time-of-flight camera according to claim 1, wherein the point spread function is stored as a matrix or lookup table in the memory.

7. The time-of-flight camera according to claim 1, wherein the point spread function is stored as Fourier transform in the memory.

8. The time-of-flight camera according to claim 1, wherein the point spread function is stored on an external device and the correction of the phase shifts or distance values is implemented on the external device.

9. A method for determining a point spread function in which a point light source and a time-of-flight camera are disposed such that a time-of-flight sensor of the time-of-flight camera detects the point light source; wherein a distance between the point light source and the time-of-flight camera and/or a beam profile of the point light source are selected such that on the time-of-flight sensor less than 5 time-of-flight pixels in a pixel row or column or a maximum of 16×16 pixels are illuminated; and wherein the point spread function is determined at least based on a subset of the time-of-flight pixels of the time-of-flight sensor.

10. The method according to claim 9, wherein the point light source is operated unmodulated.

11. The method according to claim 10, wherein modulation gates of the time-of-flight pixels of the time-of-flight sensor are driven such that charge carriers in the time-of-flight pixel are predominantly accumulated only at one integration node.

12. The method according to claim 9, wherein the point light source and the time-of-flight sensor are driven in phase with a modulation signal, and sensor difference signals are determined for at least three different phase positions.

13. The method according to claim 9, wherein for determining the point spread function at least two image frames are detected with different integration times of the time-of-flight sensor and/or different light intensities of the point light source.

Description

DRAWINGS

(1) The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

(2) The disclosure will be explained hereinafter in more detail by means of exemplary embodiments with reference to the drawings.

(3) In the drawings:

(4) FIG. 1 schematically shows a time-of-flight camera system;

(5) FIG. 2 shows a modulated integration of generated charge carriers;

(6) FIG. 3 shows a setup for determining a point spread function;

(7) FIG. 4 shows a cross section of images for determining a point spread function;

(8) FIG. 5 shows a detection of a reference scene;

(9) FIG. 6 shows a detection of an object in front of the reference scene;

(10) FIG. 7 shows measured distance values according to FIG. 6 in relation to the actual distances;

(11) FIG. 8 shows a detection of two reference surfaces with different distances;

(12) FIG. 9 shows measured distance values according to FIG. 8 in relation to the actual distances; and

(13) FIG. 10 shows a possible schematic flow of the scattered light correction in the sense of the disclosure.

(14) Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION

(15) In the following description of the preferred embodiments like reference symbols designate the same or similar components.

(16) FIG. 1 shows a measurement situation for an optical distance measurement with a time-of-flight camera as known for example from DE 197 04 496 A1.

(17) The time-of-flight camera system 1 comprises an emitting unit or an illumination module 10 comprising a light source 12 and an associated beam-shaping optics 15 as well as a reception unit or time-of-flight camera 20 with a reception optics 25 and a time-of-flight sensor 22.

(18) The time-of-flight sensor 22 has at least one time-of-flight pixel, preferably a pixel array, and is designed in particular as a PMD sensor. The reception optics 25 is typically composed of a plurality of optical elements in order to improve the imaging characteristics. The beam-shaping optics 15 of the emitting unit 10 may be formed, for example, as a reflector or lens optics. In a very simple embodiment, optionally optical elements can be dispensed with both at the reception and the emitting side.

(19) The measurement principle of this arrangement is essentially based on the fact that, based on the phase shift of the emitted and received light the time-of flight and thus the distance travelled by the received light can be determined. For this purpose, the light source 12 and the time-of-flight sensor 22 are supplied in common with a specific modulation signal M.sub.0 with a base phase position φ.sub.0 via a modulator 30. In the example shown, moreover, a phase shifter 35 is provided between the modulator 30 and the light source 12, wherein the phase shifter is able to shift the base phase φ.sub.0 of the modulation signal M.sub.0 of the light source 12 by defined phase positions φ.sub.var. For typical phase measurements, preferably, phase positions φ.sub.var=0°, 90°, 180°, 270° are used.

(20) In accordance with the adjusted modulation signal, the light source 12 emits an intensity-modulated signal S.sub.p1 with the first phase position p1 or p1=φ.sub.0+φ.sub.0var. This signal S.sub.p1 or the electromagnetic radiation is reflected in the illustrated case by an object 40 and hits due to the distance traveled 2d or the time-of-flight t.sub.L3 with a phase shift Δ.sub.φ0 with a second phase position p2=φ.sub.0+φ.sub.var+Δφ(t.sub.L) onto the time-of-flight sensor 22 as a reception signal S.sub.p2. In the time-of-flight sensor 22 the modulation signal M.sub.0 is mixed with the received signal S.sub.p2, wherein from the resulting signal the phase shift or the object distance d is determined.

(21) As the illumination source or light source 12 preferably infrared light emitting diodes or surface emitters (VCSEL) are suited. Of course, other sources of radiation in other frequency ranges are conceivable, in particular light sources in the visible frequency range come into consideration.

(22) The basic principle of the phase measurement is shown schematically by way of example in FIG. 2. The upper curve shows the timing of the modulation signal Mo with which the light source 12 and the time-of-flight sensor 22 are driven. The light reflected by the object 40 hits the time-of-flight sensor 22 with a phase shift Δφ(t.sub.L) as a reception signal S.sub.p2 according to its time-of-flight t.sub.L. The time-of-flight sensor 22 collects the photonically generated charges q over several modulation periods in the phase position of the modulation signal M.sub.0 in a first integration node Ga and in a phase position shifted by 180° in a second integration node Gb. For this steering of the charges onto the integration nodes the pixels 23 of the time-of-flight sensor 22 comprise at least two modulation gates Gam, Gbm, which steer the charges onto the first or the second integration node Ga, Gb depending on the applied modulation signals. From the difference of the charges qa, qb collected in the first and in the second integration node Ga, Gb taking into account all phase positions φ.sub.var the phase shift Δφ(t.sub.L) and thus a distance d of the object can be determined.

(23) FIG. 3 schematically shows a setup for determining a point spread function PSF. Here, the light source 112 and the time-of-flight sensor 22 can be operated unmodulated or with at least one predetermined modulation frequency. When using unmodulated light, it is advantageous if the time-of-flight sensor 22 or the pixels 23 are operated unmodulated, too. In this case, it is helpful if a constant voltage is applied to the modulation gates Gam, Gbm of the pixels 23 such that the photogenerated charges are primarily collected only in one integration node Ga, Gb.

(24) In order to determine the PSF, it is preferred that the light source 112 essentially illuminates only one single pixel 23, preferably less than 3×3 and in particular less than 5×5 pixels 23 of the time-of-flight sensor 22. In order to provide such a light spot, an aperture 150 with a sufficiently small aperture opening 152 is provided in front of the light source 112. The original light signal 10 emerging from the aperture 150 is influenced by a variety of impacts on the way to the sensor up to the detected image signal I(x), for example, by characteristics of the optical system or the optics 25 or by reflections between the sensor 22 and the optics 25. Moreover, intrinsic characteristics of the sensor 22 itself play a role, such as signal crosstalk or an electron diffusion between the pixels 23. As a result, the image signal I(x) detected at the sensor can be considered as a convolution between the incoming light I.sub.0 and a point spread function PSF, which essentially includes all characteristics of the overall system. Due to the singular illumination of one or a few pixels, the detected image signal I(x) essentially corresponds to the point spread function PSF. For determining the point spread function, preferably all pixels are evaluated. In principle, however, it is also conceivable to evaluate only a partial area around the singularly illuminated pixel.

(25) The quality of the point spread function PSF can be improved, if necessary, if several point spread functions are determined on the basis of a plurality of singularly illuminated pixels 23. For example, it is useful to additionally illuminate pixels 23 outside the optical axis in order to determine further point spread functions at these positions. On the basis of the determined point spread functions, then a point spread function can be determined which is to be used for the later corrections.

(26) Since the mentioned electron diffusion typically occurs at a diffusion rate that is significantly lower than the propagation of light, the electrons reach adjacent pixels with a time delay, so that the influence of electron diffusion also is observable as a phase shift. The point spread function PSF thus also includes complex-valued fractions. For a more accurate determination of these quantities, it is therefore advantageous to operate the light source 112 at different phase positions.

(27) Since a point spread function typically has high dynamics over several powers of ten, it is also advantageous for detecting the PSF to operate the point light source 112 with different intensities and/or the sensor 22 with different integration times.

(28) In order to compensate for dark currents, it is helpful to detect image signals I(x) both when the light source 112 is switched on and off.

(29) From the sum of all measurements, then a model of a point spread function can be generated, which is applicable to all pixels 23.

(30) Such a model can be generated according to the following considerations: Since the measured PSF is noisy and may contain, for example, artifacts that are very specific to the pixel position on the sensor, a “clean” PSF is obtained, for example, by fitting the measured PSF to a suitable model. As a model, for example
PSF({right arrow over (x)})=A({right arrow over (x)})+B(∥{right arrow over (x)}∥.sub.p.sub.B)  (1)
wherein, for example
A({right arrow over (x)})=AOexp(−s(∥{right arrow over (x)}∥p))
can be selected.

(31) Here, {right arrow over (x)} denotes the distance vector from the central pixel {right arrow over (x)}.sub.0 of the PSF in pixels and ∥{right arrow over (x)}∥.sub.p=(x.sub.1.sup.p+x.sub.2.sup.p).sup.1/p denotes the p norm of {right arrow over (x)}. p=2 would result, for example, in an exactly radial symmetrical PSF. Since the PSF need not necessarily be radially symmetric, but e.g. may be diamond-shaped, p≢2 can give better results. By a suitable choice of the p norm, anisotropies of the PSF can thus be taken into account.

(32) Since most of the light hits on the central pixel of the PSF, it is useful to add to the model a locally narrow function B(r) that reflects that fraction. This can e.g. be a Dirac Delta or a Gaussian function describing, for example, the lens blurring.

(33) For reasons of efficiency, it is advantageous to describe the PSF in the form of a spline curve, for example. In order to describe phase shifts with this PSF, the spline may, for example, have a complex-valued fraction in addition to the real fraction. This makes the PSF complex-valued, too. Suitable fitting parameters are then, for example, the values at the nodes of the splines, the normalization parameters p and pB, as well as parameters that specify the shape of B(r). Instead of storing the entire PSF, it is advantageous to just store the necessary parameters in order to generate the PSF from these parameters when initializing the software.

(34) During operation of the time-of-flight camera, then it is possible to correct the distance values with regard to scattered light influences based on the stored parameters and the PSF generated therefrom.

(35) With the aid of the setup described preferably a first image Ĩ.sub.k({right arrow over (x)}) with a short exposure time tk is detected. Specifically, the exposure time should be chosen so that none of the pixels is in saturation. In the case where modulated light is used, no pixel of the obtained raw images may be saturated.

(36) In addition, a second image Ĩ.sub.l({circumflex over (x)}) with a long exposure time t.sub.l is detected. Here, the exposure time should be chosen so that the proportion of the PSF caused by scattered light and/or crosstalk/signal crosstalk is as completely visible as possible, i.e. not affected by noise. The exposure time here is typically 1000-10000 times greater than in the first image.

(37) During the image detection either unmodulated light can be used or the light source and the sensor can be modulated in a usual manner. In the latter case the images Ĩ.sub.k({right arrow over (x)}) and Ĩ.sub.l({right arrow over (x)}) are complex-valued as usual, thus containing phase information, which reflects the time from the emission of the light up to the reception of the generated electrons at the gates of the sensor.

(38) For both images, it may be helpful to detect a series of images instead of one image and to average them to further reduce the noise.

(39) For example, in order to obtain consistent values between the first and the second image, the brightnesses (or amplitudes) are normalized with the different integration times:
I.sub.l({right arrow over (x)})={tilde over (I)}.sub.l({right arrow over (x)})/t.sub.l
I.sub.k({right arrow over (x)})={tilde over (I)}.sub.k({right arrow over (x)})t.sub.k  (3)

(40) In the images obtained the exact position of the illuminated central pixel {right arrow over (x)}.sub.0 is generally still unknown. In order to determine the position of the central pixel {right arrow over (x)}.sub.0, the first image I.sub.k({right arrow over (x)}) is binarized, for example, by means of a thresholding method, so that the bright LED spot should result in a contiguous area.

(41) The center of the contiguous area is a good estimate of the central pixel or the center point {right arrow over (x)}.sub.0 on the sensor onto which the light source is directed. This center point {right arrow over (x)}.sub.0 need not necessarily fall onto the center of a pixel, i.e. the found position for the center point {right arrow over (x)}.sub.0 need not be an integer.

(42) Now the image I.sub.k({right arrow over (x)}) of the short exposure is fitted to the model of a sharp spot. Such a model is designated for example in equation (1) with B(∥{right arrow over (x)}∥.sub.p). Here in particular

(43) $\begin{array}{cc}\left(P_{B,p_B}\right)=\arg \underset{P_{B,p_B}}{\min }\left({.Math.}_{\stackrel{.fwdarw.}{x}}{.Math.I_k\left(\stackrel{.fwdarw.}{x}\right)-B\left({.Math.\stackrel{.fwdarw.}{x}-{\stackrel{.fwdarw.}{x}}_0.Math.}_{p_B}\right).Math.}^2\right)& \left(4\right)\end{array}$
is determined, wherein PB is the parameter of the function B(r) and pB the parameter of the norm. For example, B(r)=B.sub.0exp (−br.sup.2) could be chosen, wherein then P.sub.B=(B.sub.0,b).

(44) For the numerical minimization according to equation (4) there are numerous algorithms, such as e.g. the Nelder-Mead method.

(45) In addition to PB and pB, it may give better results in equation (4) if the center {right arrow over (x)}.sub.0 of the light source is included in the optimization. Then the previously found value from the binarized image would be suitable as a starting value.

(46) Now we consider the second image I.sub.l({right arrow over (x)}) with the long exposure time. Analogously to equation (4), the image is fitted to the model of the scattered light signature, A({right arrow over (x)}) in equation (1):

(47) $\begin{array}{cc}\left(P_{A,p_A}\right)=\arg \underset{P_{A,p_A}}{\min }\left({.Math.}_{\stackrel{.fwdarw.}{x}}{.Math.I_l\left(\stackrel{.fwdarw.}{x}\right)-A\left(.Math.\stackrel{.fwdarw.}{x}-{\stackrel{.fwdarw.}{x}}_0.Math.\right).Math.}^2\right)& \left(5\right)\end{array}$

(48) If necessary, the central portion of the PSF described by B(r) can be disregarded.

(49) Analogous to the first fit, here P.sub.A are the parameters of the model function A({right arrow over (r)}). For example, a function of the form:
A({right arrow over (r)})=A.sub.0exp(−s(∥{right arrow over (r)}∥.sub.p.sub.A)+iΦ({right arrow over (r)}))  (6)
proved to be appropriate, wherein s(r) represents a (real) spline curve. The function Φ({right arrow over (r)}) describes a phase delay of the incident light spot, which may be caused, for example, by phase crosstalk/signal crosstalk between the pixels. Since this is not necessarily isotropic, it may be necessary to model Φ({right arrow over (r)}) as a two-dimensional function (e.g., a 2D spline or a 2D lookup table) instead of assuming a radially symmetric function as for s(r).

(50) The fit parameters P.sub.A are in this case A.sub.0, p.sub.A, as well as the function values of the splines at the nodes. If necessary, the position of the nodes can also be part of the fit parameters P.sub.A.

(51) For example, with the obtained parameters P.sub.A and P.sub.B, as well as the PSF model according to equation (1) it is now possible to generate an artifact-free and non-noisy PSF. Instead of storing the complete PSF, it is advantageous to store only these or other appropriate parameters from which the PSF can be generated during the initialization of the software.

(52) It is preferably provided to process the images detected with different exposure times separately from each other:
PSF({right arrow over (x)})=A.sub.1({right arrow over (x)})+A.sub.2({right arrow over (x)})+ . . .  (7)

(53) The sub-models A1 ({right arrow over (x)}), A2({right arrow over (x)}), . . . may for example correspond to different dynamic ranges of the PSF and can each be fitted separately from each other to detected images I.sub.1({right arrow over (x)}), I.sub.2({right arrow over (x)}), . . . . Based on these fit parameters, the PSF can then be summarized according to equation (7).

(54) In the foregoing, the calibration has been described with reference to a point light source with an aperture as the light source or light source system. Of course, the calibration is not limited to such a light source, but all light sources or light source systems come into consideration, which are able to generate a suitable light spot.

(55) FIGS. 4 to 9 show further methods for determining a suitable point spread function PSF. In the method according to FIG. 4, a first 3D image I.sub.1(x) of a reference scene and a second 3D image I.sub.2(x) with an object 40 in the foreground of the reference scene are detected. As already discussed, a change in the distance values known from the first 3D image I.sub.1(x) is to be expected due to systemic influences. In order to determine a point spread function suitable for the correction, then parameters of a first model PSF are varied until differences between the first and the second image, in particular distance errors, are minimal or smaller than an allowable limit. Here, preferably only the image areas or a partial area thereof are considered, in which the reference scene is visible in both images.

(56) For example, the images may be detected as shown in FIGS. 5 and 6. In a first step, a first 3D image I.sub.1(x) of a reference scene is detected (FIG. 5). As a reference scene, for example, a wall or a floor can be detected in a simple manner, but in principle any scenes with an arbitrary height profile can be detected. In the second step according to FIG. 6, an object is disposed above the reference scene, for example a hand or another object, and a second distance image I.sub.2(x) is detected. Again, the characteristics of the object are essentially uncritical. As described with reference to FIG. 4, a correction PSF can then be generated on the basis of the difference between the two images.

(57) FIG. 7 shows a variant in which the reference scene and the object are flat and disposed plane-parallel to each other. With such prior knowledge, the optimization of the PSF can possibly be simplified.

(58) Alternatively, for example, it is possible to detect only one image of a target at a sufficient distance from a flat reference scene or plane (e.g., wall, table, floor) instead of two images. In order to determine the PSF, the parameters are now varied until the reference surface behind the target is as flat as possible, or the deviations of the corrected reference surface from a plane are smaller than an allowable limit.

(59) It is particularly advantageous if the dimensions and distances of the reference scene and/or the introduced target are known in advance.

(60) FIGS. 8 and 9 show a further variant of the above procedure. The object 40 shown in FIG. 8 has a step defined in height. The height Δd=d.sub.T2−d.sub.T1 is preferably previously known. As in the abovementioned example the parameters of a PSF model are varied until the distance error is minimal or below an allowable limit.

(61) The raw images Dj (for example j=0, 1, 2, 3 corresponding to the phase positions 0°, 90°, 180°, 270° measured by the sensor are in a mathematical sense a convolution of the unknown raw images D.sub.j.sup.0 not distorted by scattered light with the PSF:
D.sub.j(x)=Σ.sub.ΔxD.sub.j.sup.0(x−Δx).Math.PSF(Δx)  (8)

(62) Interesting for further processing are the complex-valued images (8)
I(x):=(D0(x)−D2(x))+i(D1(x)−D3(x))  (9)

(63) Since the convolution is a linear operation, the same applies analogous to I(x) and the complex-valued image I.sub.0(x) not distorted by scattered light:
I(x)=Σ.sub.ΔxI.sub.0(x−Δx).Math.PSF(Δx)  (10)
or
I(x)=I.sub.0(x).Math.PSF(x)  (11)
The deconvolution is executed in the Fourier space. For this purpose, I(x) and the PSF are Fourier transformed (F[⋅]): Î(k)=F[I(x)] and (k)=F[PSF(x)]. Thus, equation 4 becomes:
Î(k)=Î.sub.0(k).Math.(k)  (12)
and thus

(64) $\begin{array}{cc}\hat{I}\left(k\right)=\frac{\hat{I}\left(k\right)}{\left(k\right)}& \left(13\right)\end{array}$
This gives the image not distorted by scattered light
I.sub.0(k)=F.sup.−1[{circumflex over (I)}.sub.0(k)]  (14)

(65) If one is interested in the correction ΔI(x):=IO(x)−I(x), i.e. the difference between the detected and the corrected image, Eq. (13) can be rearranged as follows:

(66) $\Delta \hat{I}\left(k\right):={\hat{I}}_0\left(k\right)-\hat{I}\left(k\right)=\hat{I}\left(k\right)\left(\frac{1}{\left(k\right)}-1\right)$
wherein ΔÎ(k)=F[ΔI(x)] analogous to the above treatment is the Fourier transform of the correction ΔI(x).

(67) This correction can e.g. be scaled down or reduced in resolution for performance reasons prior to the Fourier transformation and can be scaled up again to the original resolution after a scattered light correction. The correction thus obtained may subsequently be added to the detected image I(x) in order to obtain the corrected image I.sub.0(x).

(68) Due to the data reduction of course, the computational effort is reduced, too.

(69) The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.