METHOD OF CALIBRATING CAMERAS

Abstract

A method for calibrating at least one of the six-degrees-of-freedom of all or part of cameras in a formation positioned for scene capturing, the method comprising a step of initial calibration before the scene capturing. The step comprises creating a reference video frame which comprises a reference image of a stationary reference object. During scene capturing the method further comprises a step of further calibration wherein the position of the reference image of the stationary reference object within a captured scene video frame is compared to the position of the reference image of the stationary reference object within the reference video frame, and a step adapting the at least one of the six-degrees-of-freedom of a multiple cameras of the formation if needed in order to get an improved scene capturing after the further calibration.

Claims

1. A method for calibrating at least one camera(s), wherein the at least one camera is a portion of a formation, wherein each of the at least one camera(s) comprises at least two degrees of freedom, the method comprising: initial calibrating before scene capturing, wherein the initial calibrating comprises creating a reference video frame, wherein the reference video frame comprises a reference image of a stationary reference object; and capturing a scene, the capturing comprising: second calibrating, wherein the second calibrating comprises comparing a position of the stationary reference object within a captured scene video frame to a position the stationary reference object within the reference video frame; adapting the a portion of the least two degrees of freedom so as to improve scene capturing after the second calibrating; analyzing which portion of the formation have been adapted; analyzing which portion of the least two degrees of freedom have been adapted; determining corresponding values and time stamps of the adaptation of the portion of the formation and the adaptation of the portion of the at least two degrees of freedom; recognizing at least one pattern along the formation analyzing adaptation of the portion of the formation, the adaptation of the portion of the at least two degrees of freedom and the corresponding value and timestamps; and classifying the at least one pattern, wherein the classifying indicates at least one root cause in the formation.

2. The method as claimed in claim 1 further comprising repeating the analyzing of the portion cameras of the formation, the analyzing of the portion of the least two degrees of freedom, the determining the corresponding values and time stamps, the recognizing at least one pattern and the classifying of the at least one pattern.

3. The method as claimed in claim 1, wherein at least three cameras are calibrated.

4. The method as claimed in claim 3, wherein the at least two degrees of freedom comprise yaw, roll, and pitch.

5. The method as claimed in claim 1, wherein the adapting is only performed when at least one 2D image location in a set of 2D feature point image locations has changed, wherein the magnitude of the change exceeds a threshold.

6. The method as claimed in claim 1, wherein a first portion of the formation is calibrated with the initial calibration and the second calibration, wherein a second portion of the formation is calibrated with either, the initial calibration and a third calibration, or the third calibration, wherein the third calibration is performed by comparing the position of the reference image of the stationary reference object within the reference video frame of at least one neighboring camera(s) of the first portion.

7. The method as claimed in claim 6, wherein the second portion is evenly distributed within the formation.

8. The method as claimed in claim 7, wherein the comparing with the at least one neighboring camera(s) is performed with interpolation and/or extrapolation.

9. The method as claimed in claim 1, wherein the initial calibration is repeated autonomously.

10. A formation comprising: a plurality of cameras, wherein at least one camera of the formation has at least two degrees of freedom, wherein the at least one camera is initially calibrated before scene capturing, wherein a reference video frame is created during the initial calibration, wherein the reference video frame which comprises a reference image of a stationary reference object, wherein the at least one camera is second calibrated that during the scene capturing, wherein the second calibrating comprises comparing the position of the reference image of the stationary reference object within a captured scene video frame to the position of the reference image of the stationary reference object within the reference video frame, wherein the at least two degrees of freedom of the at least a portion of the formation is arranged for getting an improved captured scene.

11. (canceled)

12. A computer program stored on a non-transitory medium, wherein the computer program when executed on a processor performs the method as claimed in claim 1.

13. The formation as claimed in claim 10, wherein at least three cameras are calibrated.

14. The formation as claimed in claim 3, wherein the at least two degrees of freedom comprise yaw, roll, and pitch.

15. The formation as claimed in claim 1, wherein the arranging of the formation is only performed when at least one 2D image location in a set of 2D feature point image locations has changed, wherein the magnitude of the change exceeds a threshold.

16. The formation as claimed in claim 1, wherein a first portion of the formation is calibrated with the initial calibration and the second calibration, wherein a second portion of the formation is calibrated with either, the initial calibration and a third calibration, or the third calibration, wherein the third calibration is performed by comparing the position of the reference image of the stationary reference object within the reference video frame of at least one neighboring camera(s) of the first portion.

17. The formation as claimed in claim 6, wherein the second portion is evenly distributed within the formation.

18. The formation as claimed in claim 7, wherein the comparing with the at least one neighboring camera(s) is performed with interpolation and/or extrapolation.

19. The formation as claimed in claim 1, wherein the initial calibration is repeated autonomously.

20. The formation as claimed in claim 1, wherein the initial calibration is repeated during a specific event.

21. The method as claimed in claim 1, wherein the initial calibration is repeated during a specific event.

Description

[0022] FIG. 1 shows two linear arrays of 8 cameras each placed along the sides of a soccer field;

[0023] FIG. 2 shows example images for most left and most right cameras in an array of eight cameras as seen from the side of the soccer player field;

[0024] FIG. 3 shows a multi camera system for generating 3D data;

[0025] FIG. 4 shows pairwise rectification of cameras for an 8-camera linear array;

[0026] FIG. 5 shows schematically a set of reference feature points and a set of corresponding feature points in a 2D image plane;

[0027] FIG. 6 shows a schematic illustration of calibration; and

[0028] FIG. 7 shows method steps for classifying recognized multiple patterns for indicating different root causes of cameras getting uncalibrated.

THE INVENTION WILL BE DESCRIBED WITH REFERENCE TO THE FIGURES

[0029] It should be understood that the detailed description and specific examples are intended for illustration only and are not intended to limit the scope of the invention. These aspects and advantages will become better understood from the following description, appended claims, and accompanying drawings.

[0030] FIG. 1 shows a formation of cameras having two linear arrays of 8 cameras each placed along the sides of a soccer field. Capture cameras (thus physically present cameras) are represented by open triangles. Virtual cameras are indicated by filled triangles. The virtual cameras are physically non-existing and are only created by and existing in electronics&software. After calibration and depth estimation, virtual views may be synthesized for positions in between capture cameras or even for positions moving forward into the player field. In many cases, especially on a large sport field as a soccerfield many more cameras may be applied, e.g. 200 cameras. The cameras need not be aligned in a (virtual) straight line. For instance it can be advantageous to place a formation of cameras along a complete side of the soccerfield whereby the cameras nearby the edges (closer to the left/right goals) of the formation of cameras are portioned closer to the middle of the player field and are also more orientated in the direction of the middle of the player field. The formation can of course also be completely around the player field. Also the formation may actually be split up in several formations. For example FIG. 1 shows actually two formations of 8 cameras.

[0031] FIG. 2 shows example images captured by the left most camera and the right most camera in an 8-camera long linear array that is positioned along the side of the soccer field. As can be seen the perspective with respect to the players changes considerably from camera 1 to camera 8. The borders that can be seen at the far end of the soccer field can be used to define useful image features for calibration. Thus this features can function as stationary objects for creating one or more reference video frames. For instance the checkered panels or the stairs in between the seats may be suited for this purpose.

[0032] FIG. 3 shows a system diagram where the pairwise steps of image undistortion and rectification and disparity estimation depend on initial calibration and calibration monitoring and control which also comprises the further calibration. In view of simplicity there is only one “connection arrow” drawn from the upper “Undistort/Rectify block” to the “Initial Calibration block” and also only one “connection arrow” drawn from the “Initial Calibration block” to the upper “Undistort/Rectify block”. It will however be apparent that these connections also exist with reference to all the other “Undistort/Rectify blocks”. Input in FIG. 3 are the N camera images 11 whereby i=1 . . . N. These camera images are individually undistorted to compensate for lens distortions and focal length per camera. However, the same spatial remapping processes also rectifies each camera per camera pair. For instance, cameras 1 and 2 form the first pair in the array. Camera pairs are typically positioned spatially adjacent to each other in the array. The rectification step makes sure that, per pair, the input camera images are transformed such that they transform into a stereoscopic pair where the optical axis are aligned and rows in the image have the same position. This is essentially a rotation operation and commonly known as stereo rectification. The process is illustrated in FIG. 4.

[0033] The calibration process typically relies on a combination of the following known (see e.g. “Wikipedia”) algorithms: [0034] 1. True-range multi-lateration to determine pose based on laser measurements; [0035] 2. Perspective N-point calibration combining N scene points with known 3D position; [0036] 3. Structure-from-motion with bundle adjustment.

[0037] After initial calibration at installation, the multi-camera system may be used for different purposes. For instance, depth estimation can be performed after which virtual views may be synthesized to allow AR/VR experiences for viewers of a match. However, the system may also be used to recognize the position of an object (e.g. the foot or knee of a player) at a given moment in time and determine the 3D position based on the multiple views. Note that for the latter application, two views would be enough in theory but having many viewpoints minimizes the chance of the object of interest being occluded.

[0038] FIG. 4 shows pairwise rectification of cameras for an 8-camera linear array.

[0039] The rectification transforms each image such that the optical axes, indicated with arrows (R1, R1a; R2, R2a; R3, R3a; R4, R4a) become pairwise parallel, indicated with arrows (G1, G1a; G2, G2a; G3, G3a; G4, G4a) and orthogonal to the line (dotted) that connects both cameras. The rectification allows for easier disparity estimation and depth calculation.

[0040] FIG. 5 shows schematically a set of reference feature points and a set of corresponding feature points in a 2D image plane. The correspondence relation is determined via motion estimation and illustrated via the dotted line. As can be seen a corresponding point in a new frame cannot always be found since occlusion may occur or motion estimation may fail due to image noise.

[0041] FIG. 6 shows a schematic illustration of calibration by showing an example of a changing camera (camera 1 in this case). Original features are detected in a reference frame and their image position is represened by open circles. Corresponding feature points are thereafter determined via motion estimation in a first frame and in a second frame. These corresponding points are represented as closed circles. As shown the points do not change position for frames 1 for both cameras 1 and 2 and for frame 2 for camera 2. However, for camera 1 points 1, 2 and 3 change position. It can therefore be concluded that camera 1 is no longer calibrated. Notice that for reference point 4 which is only visible in camera 1, a corresponding feature point was never found. This point is denoted to be ‘invalid’ and therefore not taken into account when calculating an error metric on the basis of which we conclude that a camera is no longer calibrated (see equation further on).

[0042] Thus the 3D scene points are first projected in a reference frame (represented by the open circles). Correspondence is estimated for these points with further frames (represented by closed circles) using image based matching. Calibration status for further frames is judged OK when a given fraction of corresponding feature points don't change position. This is the case for both camera 1 and camera 2 in frame 1. However, in frame 2, camera 1 shows a rotation error since three points are displaced compared to the reference points.

[0043] FIG. 7 show the steps for classifying recognized patterns (in six-degrees-of-freedom disturbances of a camera) for indicating different root causes of cameras getting uncalibrated.

[0044] In step A it is analyzed which cameras of a formation of cameras have been adapted caused by an external disturbance, like e.g. wind.

[0045] In step B for each adapted camera it is analyzed which of the six-degrees-of-freedom is/are adapted. For instance in step A it has been determined that for a first camera only the orientational direction “yaw” has been adapted and for a second camera both orientational directions “Yaw” and “Roll” and the translational direction “left-right” have been adapted.

[0046] In step C the values and time stamps of the adaptions are determined. For instance for the said first camera the value of the “Yaw” is 0.01° and for the said second camera the value of the “Yaw” and “Roll” are respectively 0.01° and 0.02° and the value of “left-right” is +12 mm. (E.g. +12 mm is 12 mm to the right, while −12 mm would be 12 mm to the left.) The time stamps for both cameras is for instance 13:54 h+10 sec.+17 ms. (In principal the timestamps for both cameras may also differ.) Also with an orientation e.g. “Pitch” a sign can be dedicated to the value, e.g. −0.01° but alternatively also values of the angle could be indicated without a sign if full use is made of 360° representation that is to say−0.01° can also be indicated as +359.99°.

[0047] In step D multiple patterns along the formation of cameras are recognized along the formation of cameras by analyzing the information resulted from steps A, B, and C. E.g. with reference to a first timestamp all values of the adaptations related to “Yaw” are registered as a first pattern of camera deviations, all values of the adaptations related to “Roll” are registered as a second pattern of camera deviations, etc.

[0048] In step E the recognized multiple patterns are classified whereby the classification indicate different root causes for camera disturbances in one or more of the six-degrees-of-freedom of cameras in the formation. Examples for the classification are: “wind”, “mechanical stress”, “temperature effect”, etc. These classifications are e.g. determined by knowledge resulted from earlier measurements. For example the said first pattern is compared with all possible classifications. It is not necessary that the first pattern has only a “hit” if the corresponding classification exactly matches, a high resemblance is sufficient. Use can be made of all kind of (known) pattern recognition methods. Of course if no “hit” at all is found then the pattern can be classified as “unknown cause”

[0049] Optionally in step F, step E takes multiple analyzing sessions of steps A, B, C, and D into account. This gives additional possibilities for classifying the patterns. E.g. if a calibration issue comes back regularly for example for only two cameras in a large formation of cameras, the classifier may then output: “recurring group disturbance”.

[0050] A suitable error metric for detecting calibration problems is the sum of distances between corresponding positions for all valid feature points:

[00001]$e\equiv \frac{1}{N_{\mathrm{valid}}}\underset{i=1}{\overset{N_{\mathrm{valid}}}{.Math.}}.Math.x_{\mathrm{ref},i}-x_i.Math.$

[0051] where x.sub.ref,i is the reference image feature point, x.sub.i is the matched image feature point and N.sub.valid is the number of features that is matched succesfully from the reference video frame to the new video frame. A simple test now evaluates:

e<T,

where T is a pre-set threshold [pixel]. This threshold is typically set to a magnitude between 0.1 and 5 pixel. Once the test fails, the status of the camera involved is set to ‘uncalibrated’.

[0052] In case the calibration test fails for a single camera but not for the other cameras, it can be concluded that a local disturbance occurred only affecting a single camera. Under the (realistic) assumption that only camera orientation has changed the camera can be re-calibrated in real-time.

[0053] First, observe that the distance r.sub.i from the camera to each 3D scene point i must remain constant since neither the scene point nor the camera position change. The camera coordinates of point i can be calculated using:

[00002]$z_{c,i}=\frac{{\mathrm{fr}}_i}{\sqrt{u^2+v^2+f^2}}$$x_{c,i}=\frac{{\mathrm{uz}}_{c,i}}{f}$$y_{c,i}=\frac{{\mathrm{vz}}_{c,i}}{f}$

where f is the focal length [pixel] and u, v are image coordinates [pixel]. This gives calibration points in both world space and camera space. The well-known Kabsch algorithm (see e.g. “Wikipedia”) can then be used to estimate the rotation matrix that takes a point in world space and transforms it to camera space. The new rotation matrix is only accepted as part of an updated view matrix when after projection of the 3D scene points into the image the re-projection error e decreases. Note that even then, the camera status may still evaluate to ‘uncalibrated’. This happens when e T.

[0054] It is also useful to detect a slowly changing trend in camera orientation. If such a trend is present then the root-cause is likely “mechanical stress or temperature” when the re-projection error e exceeds threshold T.

[0055] A linear array of five cameras can show the following spatial pattern of errors:

e.sub.1<T,e.sub.2<T,e.sub.3T,e.sub.4T,e.sub.5<T.

[0056] As can be seem, the problem is still localized (at cameras 3 and 4). It could still be the case that a “local disturbance” (close to cameras 3 and 4) causes the problem. Real-time re-calibration of both cameras is the sufficient action to take. If the problem comes back regularly for only these two cameras, the classifier will output: “recurring group disturbance”. A suitable action is then to exclude the image data from these two cameras and base the view synthesis and/3D analysis on the other cameras that still have the calibrated status. It may be that a “recurring group disturbance” is related to a specific location in the stadium. By logging re-projection errors over an entire game it can be analyzed which spatial locations in the stadium systematically cause most instabilities. For a next match, cameras may be placed at a different location.

[0057] A strong wind, cheering people (stadium vibration) or a passing truck can change the orientation of cameras. These root-causes typically affect all cameras but may only have a temporary influence on the orientations of all cameras. It may be that these types of disturbances will temporarily increase the re-projection error for all cameras but that due to elasticity in the system the re-projection error will decrease again (e.g., when people have stopped cheering). In this situation where for all cameras the re-projection error passes the threshold by a small amount we can choose to take no action at all, i.e. we tolerate this error and pass the information downstream to the view synthesis and analytics module. Performing the same analysis after the game on logged data will again provide valuable insights on whether or not to make mechanical changes to the camera configuration (e.g. increasing mass of each camera or changing the mounting point onto the existing infrastructure of the stadium). A spatio-temporal analysis (e.g. Fourier transform, spectral analysis) can be used to classify which event occurred.

[0058] While the spatio-temporal pattern in the re-projection error is likely informative enough to do a status classification, it may be even more informative to observe signals of orientation changes for cameras as a function of space and time. This can reveal the specific rotational motion that the camera rig goes through. For instance, it may be that the motion is predominantly around the horizontal axes in case mechanical flexibility is greater vertically. Such data can hence provide ideas for improved camera mounting for future captures.

[0059] Some applications require that every camera in the camera array is oriented in real-time towards the action that takes place on the playing field. A control algorithm then continuously controls the servos of each camera to best direct each camera to the action. In this situation, the rotation matrix estimate must be updated in real-time for each camera. Doing status classification is more complex in this case. To classify the status, the camera orientation is continuously predicted using the servo control signal. The orientation prediction is then used to update the view matrix and the re-projection error is evaluated for the updated view matrix. External effects, such as wind, can still be classified via the re-projection error. However, it is also possible to use the pan-tilt system to correct for the error introduced by e.g. the wind. In that case, the control signal is generated to not only to direct the cameras to the action but also has a component that corrects for wind in real-time. The estimation of, and the control for, the wind effect can be done using a Kalman filter. (See e.g. “Wikipedia”.)

[0060] It is emphasized that the invention is not only limited with regard to sports events but can be applied in many different applications. E.g. it can be applied in shopping-centres, market places, automotive industry etc. It is also not limited to outdoor activities but can e.g. also be useful for real estate brokers, for medical staff in a hospital etc.

[0061] With reference to the automotive industry the invention can also be useful applied for instance on a car. For instance an array of cameras can be installed at the rear bumper. These cameras may replace inside and/or outside mirrors. For a car with a camera array built-in the back of the car looking backwards the road will constantly move through the images. However, if the speed of the car is known, the road part of the images can be made stationary. The situation is then the same as a camera array looking onto a stationary sports field. A car vibration, e.g. due to a gravel road will cause small changes in camera rotation at a high temporal frequency. These can be detected, and corrected for using the propped approach. Recognition/classification of road-type (smooth highway versus gravel road) will help the real-time calibration process. If the car takes a turn into another road then an initial calibration is started. Once the car is driving on the new road, the reference frame is continuously being updated (tracked) when new road becomes visible during driving.

[0062] With reference to the tenth embodiment (claim 10), STEP F: in the context of a car, the (earlier mentioned) low-frequency event is the image of the road translating below the car and the (earlier mentioned) high-frequency event is the vibration caused by the gravel road or bumps in the road. Note that gravel roads and bumps in the road are expected to cause also relative orientation/position changes between cameras mounted into the bumper. It is expected that these relative changes are small when driving on a smooth highway. Since the car bumper will have elastic properties a calibration can be temporary off (due to a bump in the road) and then recover from itself (due to the flexible structure of the car bumper).

[0063] It is further noted that all kind of camera formations can be used, it is for instance also possible to use drones whereby each drone comprise one or more cameras. Big advantage of drones is that the mutual positions of the camera can be very easily adapted. The term “formation” is considered to be similar to the term “arrangement” or “group”, etc.

[0064] Amongst numerous VR/AR-applications (with e.g. a computer or VR-goggles), the invention can also be used both in 2D or 3D applications.

[0065] The invention can generally be summarized by the following

Embodiments (1-12)

[0066] 1. A method for calibrating at least one of the six-degrees-of-freedom of all or part of cameras in a formation positioned for scene capturing, the method comprising a step of initial calibration before the scene capturing, wherein the step comprises creating a reference video frame which comprises a reference image of a stationary reference object, and wherein during scene capturing the method further comprises: [0067] a step of further calibration wherein the position of the reference image of the stationary reference object within a captured scene video frame is compared to the position of the reference image of the stationary reference object within the reference video frame, and [0068] a step of adapting the at least one of the six-degrees-of-freedom of a multiple cameras of the formation if needed in order to get an improved scene capturing after the further calibration.

[0069] 2. A method according to embodiment 1, wherein the “at least one” is “three”.

[0070] 3. A method according to embodiment 2, wherein the three degrees of freedom are yaw, roll, and pitch.

[0071] 4. A method according to any previous embodiment, wherein the step of adapting is only performed when the outcome of the comparison exceeds a threshold.

[0072] 5. A method according to any previous embodiment, wherein only a first part of the cameras is calibrated with the initial calibration and the further calibration and wherein the remaining second part is only calibrated with either: [0073] only the initial calibration and an alternative calibration, or [0074] only an alternative calibration, wherein the alternative calibration of a camera belonging to the second part is performed by comparing with one or more neighboring cameras of the first part.

[0075] 6. A method according to embodiment 5, wherein the second part of cameras is evenly distributed along all the cameras in the formation.

[0076] 7. A method according to embodiment 6, wherein the comparing with the one or more neighboring cameras is performed with interpolation and/or extrapolation techniques.

[0077] 8. A method according to any previous embodiment, wherein the initial calibration is continuously repeated autonomously or at a specific event.

[0078] 9. A method according to any previous embodiment, wherein the method further comprises: [0079] step A: analyzing which cameras of the formation have been adapted [0080] step B: analyzing which of the six-degrees-of-freedom of the adapted cameras have been adapted [0081] step C: determining the corresponding values and time stamps of the adaptations [0082] step D: recognizing multiple patterns along the formation of cameras by analyzing the information resulted from steps A, B, and C, and [0083] step E: classifying the recognized multiple patterns whereby the classification indicate different root causes for camera disturbances in one or more of the six-degrees-of-freedom of cameras in the formation.

[0084] 10. A method according to embodiment 9, wherein the method further comprises: [0085] step F: in step E taking into account multiple analyzing sessions of steps A, B, C, and D.

[0086] 11. A formation of cameras positioned for scene capturing wherein at least one camera has at least one of six-degrees-of-freedom wherein the at least one camera is initially calibrated before the scene capturing, wherein during the initial calibration a reference video frame is created which comprises a reference image of a stationary reference object, and wherein during the scene capturing the at least one camera is further calibrated wherein the position of the reference image of the stationary reference object within a captured scene video frame is compared to the position of the reference image of the stationary reference object within the reference video frame, and wherein the at least one of the six-degrees-of-freedom of the at least one camera of the formation is adapted if needed for getting an improved captured scene.

[0087] 12. A computer program comprising computer program code means adapted, when said computer program is run on a computer, to implement the method of any of embodiments 1 to 10.

and more particularly by the appended CLAIMS.

[0088] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0089] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.