SYSTEM AND METHOD FOR CONTROLLING A LIGHT SOURCE FOR ILLUMINATING A SCENE OF INTEREST

Abstract

The invention relates to a for controlling a light source, the method using (a) at least one pose estimate of a camera configured to capture one or more images of a scene of interest which comprises at least one landmark, as said light source is operated to emit light which illuminates said scene of interest, (b) a landmark map comprising at least 3D location information of a plurality of landmarks comprising the at least one landmark in the scene of interest, (c) an illumination model describing a relationship between an emission illumination power and a reflection illumination power, wherein said emission illumination power is the power of light emitted by the light source to illuminate said scene of interest, and said reflection illumination power is the illumination power of light reflected by one or more landmarks in said scene of interest and received by the camera, wherein the method comprises the following steps: (1) determining, for at least one of the plurality of landmarks, at least one emission illumination power of light to be emitted by the light source, and an illumination time during which the light source should be operated to emit light which has an emission illumination power which is equal to the at least one emission illumination power, using (i) the at least one pose estimate of the camera, (ii) the 3D location information of the at least one of the plurality of landmarks, (iii) the illumination model; and (2) operating the light source to emit light which has an emission illumination power which is equal to the at least one emission illumination power, for a time period which is equal to the determined illumination time course; and (3) updating the illumination model parameters, wherein the illumination model parameters which are updated comprise reflectivity parameters of at least one landmark.

Claims

1. Method for controlling a light source, the method using (a) at least one pose estimate of a camera configured to capture one or more images of a scene of interest which comprises at least one landmark, as said light source is operated to emit light which illuminates said scene of interest, (b) a landmark map comprising at least 3D location information of a plurality of landmarks comprising the at least one landmark in the scene of interest, (c) an illumination model describing a relationship between an emission illumination power and a reflection illumination power, wherein said emission illumination power is the power of light emitted by the light source to illuminate said scene of interest, and said reflection illumination power is the illumination power of light reflected by one or more landmarks in said scene of interest and received by the camera, wherein the method comprises the following steps: (1) determining, for at least one of the plurality of landmarks, at least one emission illumination power of light to be emitted by the light source, and an illumination time during which the light source should be operated to emit light which has an emission illumination power which is equal to the at least one emission illumination power, using (i) the at least one pose estimate of the camera, (ii) the 3D location information of the at least one of the plurality of landmarks, (iii) the illumination model; and (2) operating the light source to emit light which has an emission illumination power which is equal to the at least one emission illumination power, for a time period which is equal to the determined illumination time course; and (3) updating the illumination model parameters, wherein the illumination model parameters which are updated comprise reflectivity parameters of at least one landmark.

2. The method according to claim 1 wherein the step (3) of updating the illumination model parameters comprises, updating the illumination model parameters based on a deviation between the predicted received illumination power and measured received illumination power.

3. Method according to claim 1, wherein the at least one emission illumination power is a solution to an optimization problem for energy of the emitted light delivered in the predefined illumination time course, so that said at least one emission illumination power is an optimized emission illumination power.

4. Method according to claim 1, wherein the at least one emission illumination power is a solution to an optimization problem for the uncertainty of a posterior first pose estimate, so that said at least one emission illumination power is an optimized emission illumination power.

5. Method according to claim 1, wherein the illumination model is configured to model a non-isotropically emitting light source.

6. Method according to claim 1, wherein the at least one pose estimate comprises a first pose estimate, and wherein the determining of the at least one emission illumination power comprises determining a first emission illumination power by: (a) determining distances between the first pose estimate and the 3D location of the plurality of landmarks, (b) sorting the distances in an ascending order or descending order, (c) choosing an M-th distance from the sorted distances, and (d) using the M-th distance for determining the first emission illumination power using at least the illumination model.

7. Method according to claim 1, wherein the at least one emission illumination power is determined using a constrained optimization algorithm with a predefined illumination time course, wherein the constrained optimization algorithm is configured to extremize a cost function while fulfilling constraints.

8. Method according to claim 7, wherein the constrained optimization algorithm is configured to minimize or maximize the cost function, by varying at least the at least one emission illumination power.

9. Method according to claim 1, wherein the illumination model comprises illumination model parameters, wherein at least one of the illumination model parameters is a stochastic parameter, wherein determining the at least one emission illumination power involves stochastically propagating illumination model parameter uncertainty through the illumination model.

10. Method according to claim 1, wherein the at least one pose estimate comprises a first pose estimate and wherein the first pose estimate comprises a first pose estimate uncertainty, wherein a first emission illumination power is determined together with a posterior first pose estimate uncertainty of the camera, which posterior first pose estimate uncertainty is determined, together with the first emission illumination power, using at least (i) the first pose estimate and the first pose estimate uncertainty, (ii) the illumination model, and (iii) a localization model for determining a pose of the camera using positions of landmarks in an image acquired by the camera at the pose, wherein the first emission illumination power is determined in such a way that the posterior first pose estimate uncertainty is below a predefined posterior uncertainty threshold.

11. Method according to claim 10, wherein the first emission illumination power is determined together with a second pose estimate uncertainty of the camera, which second pose estimate uncertainty is determined, together with the first emission illumination power, by additionally using a movement model of the camera, wherein the first emission illumination power is determined in such a way that the second pose estimate uncertainty, obtained by forward-projecting the posterior first pose estimate uncertainty using the movement model to a time tt.sub.2, tt.sub.2>tt.sub.1, at which the camera is configured to capture a subsequent image, is below a predefined uncertainty threshold.

12. Method according to claim 1, wherein the first emission illumination power is determined by solving the following constrained optimization problem $\underset{{\mathrm{EE}}_{\mathrm{TTTT}}}{\min }{\mathrm{EE}}_{\mathrm{TTTT}}\mathrm{ss}.\mathrm{tt}.{.Math.}_{\mathrm{pp}}\left({\mathrm{EE}}_{\mathrm{TTTT}}\right)<{}_{\mathrm{pp},\mathrm{mmmmmm}},$ wherein EE.sub.TTTT is the energy of the emitted light delivered in the predefined illumination time course, is a matrix norm, .sub.pp(EE.sub.TTTT) is a covariance matrix relating to the posterior first pose estimate uncertainty, which covariance matrix depends on the energy of the emitted light, wherein the posterior first pose estimate is determined based on landmarks which are detectable in the image in case the emitted light has energy EE.sub.TTTT, and wherein .sub.pp,mmmmmm is the predefined posterior uncertainty threshold, so that the said first emission illumination power is a first optimized emission illumination power.

13. Method according to claim 12, wherein the constrained optimization algorithm is relaxed to an unconstrained optimization algorithm, $\underset{{\mathrm{EE}}_{\mathrm{TTTT}}}{\min }{\mathrm{EE}}_{\mathrm{TTTT}}+{.Math.}_{\mathrm{pp}}\left({\mathrm{EE}}_{\mathrm{TTTT}}\right),$ wherein is a scaling parameter tending to infinity, so that said emission illumination power is an optimized emission illumination power.

14. Method according to claim 7, wherein a first emission illumination power is determined by solving the following constrained optimization problem, $\underset{{\mathrm{EE}}_{\mathrm{TTTT}}}{\max }{\mathrm{EE}}_{\mathrm{TTTT}}\mathrm{ss}.\mathrm{tt}.{\mathrm{EE}}_{\mathrm{RRTT},\mathrm{IImm}}<{\mathrm{EE}}_{\mathrm{RRTT},\mathrm{mmmmmm}}\mathrm{IIII},$ wherein EE.sub.TTTT is the energy of the emitted light delivered in the predefined illumination time course, IIII is an index over at least a subset of the plurality of landmarks, wherein EE.sub.RRTT,llmm is a reflection illumination energy corresponding to the IIII-th landmark, and EE.sub.RRTT,mmmmmm is a reflection illumination energy at which the camera saturates, so as to provide a first optimized emission illumination power.

15. Method according to claim 1, wherein the illumination model is embodied as follows, ${\mathrm{PP}}_{\mathrm{RRTT}}=\frac{1}{4{\mathrm{dd}}^2}{\mathrm{PP}}_{\mathrm{TTTT}}{\mathrm{RR}}_{\mathrm{AA}}{\mathrm{AA}}_{\mathrm{RR}}\frac{1}{{\mathrm{dd}}^2}\frac{{\mathrm{ff}}^2}{4{\mathrm{NN}}^2},$ wherein PP.sub.RRTT is the reflection illumination power, PP.sub.TTTT is the emission illumination power, ff is the focal length of the camera, NN is the f-number of the camera, RR.sub.AA is a landmark reflectivity of a reflector landmark, wherein at least one landmark of the plurality of landmarks is embodied as said reflector landmark, AA.sub.RR is a projected surface area of said reflector landmark as viewed from the light source, and dd is the distance between the light source and said reflector landmark.

16. Method according to claim 1, wherein the determining of the at least one emission illumination power comprises comparing a predefined threshold reflection illumination power to a predicted received illumination power using the illumination model, wherein the at least one emission illumination power is set in such a way that a corresponding at least one predicted received illumination power is equal to or greater than the predefined threshold reflection illumination power.

17. Method according to claim 1, wherein the at least one pose estimate comprises a plurality of pose estimates and wherein the at least one emission illumination power comprises a plurality of emission illumination powers, the plurality of pose estimates and emission illumination powers relating to a planned and/or predicted movement of the camera, and adapting the planned and/or predicted movement based on an output of the constrained optimization algorithm.

18. Method according to claim 17, wherein the at least one emission illumination power is determined by solving the following constrained optimization problem $\underset{\left(\begin{array}{c}\\ E_{\mathrm{TX},1}\\ E_{\mathrm{TX},2}\\ .Math.\\ E_{\mathrm{TX},]}\end{array}\right)}{\min }\underset{i=1}{\stackrel{J}{L}}{E}_{\mathrm{TX},i}s.t.g\left(\left(\begin{array}{c}\\ E_{\mathrm{TX},1}\\ E_{\mathrm{TX},2}\\ .Math.\\ E_{\mathrm{TX},J}\end{array}\right)\right)0,$ wherein JJ is a natural number denoting that emission illumination power is to be minimized for JJ subsequent images and wherein emission illumination energies EE.sub.TTTT,mm, EE.sub.TTTT,mm being the energy of the i-th emission of light, for the JJ subsequent images are arranged in a vector, and wherein the cost function is embodied as sum over the JJ emission illumination energies, and wherein gg is a constraint function embodied as vector-valued function or as scalar-valued function, and taking the vector comprising the JJ emission illumination energies as input, so that said emission illumination power is an optimized emission illumination power.

19. A non-transitory storage medium containing computer program product comprising instructions which when executed by a computer, cause the computer to carry out a method according to claim 1.

20. Assembly, comprising (a) a light source, (b) a camera, (c) a plurality of landmarks, and (d) a controller which is configured to carry out a method according to claim 1.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0063] Exemplary embodiments of the invention are disclosed in the description and illustrated by the drawings in which:

[0064] FIG. 1 shows a schematic depiction of the method, according to an embodiment of the present invention, for controlling a light source; and

[0065] FIG. 2 shows a schematic depiction of a drone comprising a light source and a camera, wherein the drone is configured to fly in an indoor environment, wherein landmarks are arranged at a plurality of positions in said indoor environment.

DETAILED DESCRIPTION OF DRAWINGS

[0066] FIG. 1 shows a schematic depiction of the method, according to an embodiment of the present invention, for controlling a light source. The method receives at least one pose estimate 1, a landmark map 2, an illumination model 3 and a predefined threshold illumination power 4 as input. Using these inputs, at least one optimized emission illumination power 6 and an illumination time course 6 are determined 5.

[0067] The at least one pose estimate 1 may relate to a 3D position and orientation of a camera, which camera is configured to capture images of a scene of interest. The scene of interest may be in an indoor environment, e.g., a warehouse in which a drone carrying at least a camera and a light source may need to navigate. In case the at least one pose estimate comprises a plurality of pose estimates, these pose estimates may be related to 3D positions and orientations of the camera at times at which a corresponding plurality of images is captured by the camera. A plurality of pose estimates may relate to poses of the camera at future times, e.g., to a planned and/or estimated future motion of the camera. Future motion may be at least partly inferred from past motion, e.g., using a Kalman filter for extrapolation, and/or may be obtained from control input. A drone carrying the camera may have inertia, which inertia may prohibit arbitrarily fast movement changes.

[0068] A drone carrying the camera and the light source may operate in an indoor environment equipped with landmarks, which landmarks may be used by the drone to determine its current location in the indoor environment. In prior art, the problem of 3D pose determination of a calibrated camera using images of such external landmarks captured by said calibrated camera is known as perspective-n-point problem. If sufficiently many landmarks with known position in a world coordinate system are visible in an image captured by the camera (two landmarks may need to be visible in case additional orientation information of the projected landmarks is available, or generally three landmarks may need to be visible), a pose of the calibrated camera may be determined using known algorithmic solutions. A 3D pose of the camera may also be determined using an inertial measurement unit (IMU) attached to the drone, using a known coordinate transformation between the IMU and the camera, or by a combination of inertial measurements and computer-vision-based pose determination algorithms, potentially combined using a Kalman filter. A Kalman filter may also be used for determining a pose. Pose estimates may be provided by a Kalman filter through extrapolation.

[0069] For a computer-vision-based pose determination of the calibrated camera, e.g., using well-known algorithmic solutions to the perspective-n-point problem, sufficiently many landmarks may need to be visible in an image captured by the calibrated camera. In order to facilitate landmark detection and visibility in images, at least some landmarks may be embodied as retroreflectors, which retroreflectors may be installed at known positions in the scene of interest. In case a light source is mounted in vicinity to the camera on the drone, and said light source is used for emitting light, the retroreflector may be clearly visible in an image captured by the camera. The landmark map 2 may comprise information on the positions on landmarks in a world coordinate system. The landmark map 2 may also be determined using a simultaneous-localization-and-mapping (SLAM) algorithm carried out while the drone carrying the camera and the light source moves about the environment.

[0070] The illumination model 3 describes how much of the power emitted by the light source arrives at the camera after reflection by a landmark, e.g., a retroreflector. The illumination model 3 is therefore preferentially a physical model, which physical model may describe the power losses occurring between light emission by the light source and light reception by the camera. The illumination model 3 may therefore, e.g., need to reflect whether the light source is an isotropically or non-isotropically emitting light source, whether a landmark reflects diffusely or narrowly etc. Some parameters of the illumination model 3 may be only known approximately. In this case, these parameters may be estimated during operation of the drone carrying the camera and light source, and/or may be continuously tracked in case they are changing over time. The illumination model 3 may comprise information on the distance between light source and landmark, in particular embodied as retroreflector, and between landmark and camera. Some parameters of the illumination model 3 may therefore differ between different landmarks, e.g., due to different distances of landmarks to the light source.

[0071] Using the illumination model 3, it may be determined how muchvirtuallyemitted light power reaches the camera, specifically the subset of pixels of an image sensor of the camera onto which the corresponding landmark is imaged. In order for the captured feature (image of the illuminated landmark) to be detectable in a reliable and accurate manner from an image, the captured feature needs to be sufficiently bright. The output of the illumination model 3 may therefore be virtually compared to the predefined threshold illumination power 4, which predefined threshold illumination power 4 may, e.g., relate to a noise floor of the camera. This way, an optimized emission illumination power of power emitted by the light source may be determined which may guarantee that a specific landmark can be detected in a reliable manner in an image.

[0072] For pose determination of the camera using projections of landmarks into a captured image, a specific set of landmarks may be chosen. The chosen set of landmarks may be chosen in such a way that reliable pose determination can be achieved, e.g., landmarks whose projections into an image are too close to each other may be disregarded. The optimized emission illumination power may be set in such a way so as to facilitate detection of the chosen set of landmarks in a captured image. The optimized emission illumination power is determined based on the at least one pose estimate 1. At any given camera pose, it may not be possible to see all landmarks with the camera. In the optimization of power for the emission illumination power, such landmarks which are not visible may be disregarded. Landmarks which are currently not visible but may become visible at a later timepointe.g., using knowledge of a planned and/or predicted movement of a drone carrying the cameramay, however, be included in the emission illumination power optimization process, while landmarks which are currently visible but may disappear at a later timepointe.g., using knowledge of a planned and/or predicted movement of the dronemay potentially be disregarded in the emission illumination power optimization process, e.g., in case the emission illumination power is determined for a future pose of the camera.

[0073] Determining 5 the optimized emission illumination power 6 may be carried out using an optimization algorithm, in particular a constrained optimization algorithm. Once a set of landmarks is determined, distances between the camera and the chosen set of landmarks may be determined using the at least one pose estimate and the landmark map. The determined distances can then be used for parametrizing the illumination model 3, which illumination model 3 is used for providing a link between the emission illumination power and a received illumination power received by the camera (the term received illumination power corresponds to the term reflection illumination power). The optimized illumination power then provides a set of features in an image captured by the camera, which set of features can be used for pose determination. The pose estimate provided as input may in this way be transformed into a determined pose, i.e., into a posterior pose estimate, e.g., by using a Kalman filter.

[0074] FIG. 2 shows a schematic depiction of a drone comprising a light source 7 and a camera 8, which drone is flying in an indoor environment 13. Landmarks 9, in particular embodied as retroreflectors, are arranged at a plurality of positions in the indoor environment 13, which indoor environment 13 may be a scene of interest. The landmarks 9 may be mounted on a ceiling in the scene of interest 13. Instead of being mounted to a ceiling in the scene of interest 13, the landmarks 9 may also be an integral part of the scene of interest 13 and may also be located on walls and/or a floor of the scene of interest 13. At any given pose of the drone, some landmarks 9 may be visible to the camera 8in FIG. 2 indicated by lines between the landmarks 9 and the camera 8while other landmarks 9 may not be visible to the camera 8. The positions of the landmarks 9 may be known in a world coordinate system 10, and the current location of the drone may be expressed in a drone coordinate system 11, wherein a coordinate transformation 12 may be known between the world coordinate system 10 and the drone coordinate system 11. In case the camera 8 and the light source 7 are mounted rigidly to the drone, the pose of the camera 8 and of the light source 7 can be related to the world coordinate system 10 using the drone coordinate system 11. The current position of the drone can be determined using image(s) of the scene of interest 13, specifically of the landmarks 9 having known positions. Alternatively, or in addition, the drone may be equipped with an inertial measurement unit, which inertial measurement unit may be also used for pose determination. The light source 7 may be an isotropically emitting light source, or it may be a directional light source emitting in a non-isotropic manner. Light source 7 and camera 8 are ideally close to each other, specifically in case the landmarks 9 are embodied as retroreflectors. The camera 8 may also be mounted on top of the drone, i.e., next to the light source 7.