System and Method for Recalibrating a Projector System

Abstract

A system and method for recalibrating a projector system. The system includes one or more cameras, a projector, and at least one processor. The at least one processor is configured to execute stored instructions to project one or more patterns on to a work surface via the projector, capture images of the projected one or more patterns via the one or more cameras, perform analysis on the captured images of the one or more patterns, determine projector calibration parameters based at least in part on the performed analysis on the captured images, and recalibrate the projector system such that an image is projected onto the work surface by the projector at a correct position based on the determined projector calibration parameters.

Claims

1.-20. (canceled)

21. A recalibratable projector system, comprising: one or more cameras; a projector; and at least one processor configured to execute stored instructions to: project one or more patterns on to a work surface via the projector; capture images of the projected one or more patterns via the one or more cameras; perform analysis on the captured images of the one or more patterns; determine projector calibration parameters based at least in part on the performed analysis on the captured images; and recalibrate the projector system such that an image is projected onto the work surface by the projector at a correct position based on the determined projector calibration parameters.

22. The recalibratable projector system of claim 21 further comprising one or more of: (i) at least one laser scanner and (ii) at least one surface profiling sensor, wherein the one or more cameras, the at least one scanner, the at least one surface profiling sensor, and the projector are configured to profile the work surface.

23. The recalibratable projector system of claim 21, wherein the one or more cameras are mechanically affixed to the recalibratable projector system and calibrated relative to each other and relative to the projector.

24. The recalibratable projector system of claim 23, wherein the calibration of the one or more cameras is performed only once and is a known parameter.

25. The recalibratable projector system of claim 23, wherein the one or more cameras are mechanically affixed to the recalibratable projector system via one or more of: (i) a screw, (ii) a fastener, and (iii) an adhesive.

26. The recalibratable projector system of claim 21, wherein the one or more patterns includes specific geometric shapes.

27. The recalibratable projector system of claim 21, wherein the at least one processor is further configured to profile the work surface and wherein the profiling of the work surface and recalibration of the projector system are performed simultaneously.

28. The recalibratable projector system of claim 21, wherein the determination of the projector calibration parameters includes calculating correspondences between points from the one or more cameras and the projector.

29. The recalibratable projector system of claim 21, wherein the determination of the projector calibration parameters includes solving at least one equation system for unknown parameters.

30. The recalibratable projector system of claim 29, wherein the at least one equation system is solved based on a Levenberg-Marquardt algorithm.

31. The recalibratable projector system of claim 29, wherein the determination of the projector calibration parameters includes obtaining intrinsic parameters and extrinsic parameters.

32. The recalibratable projector system of claim 31, wherein the intrinsic parameters include one or more of: (i) projector focal lengths in an x-direction and a y-direction, (ii) principal point in the x-direction and the y-direction, and (iii) skew coefficient between an x-axis and a y-axis.

33. The recalibratable projector system of claim 31, wherein the extrinsic parameters include one or more of: (i) translation vector between the projector and the one or more cameras and (ii) rotation matrix between the projector and the one or more cameras.

34. The recalibratable projector system of claim 21, wherein the one or more patterns includes the image itself.

35. The recalibratable projector system of claim 21, wherein the one or more cameras includes an infrared camera.

36. The recalibratable projector system of claim 21, wherein the image is a blueprint of a construction-related task.

37. The recalibratable projector system of claim 21, wherein the projector is a position enabled projector.

38. A method for recalibrating a projector system, comprising the steps of: projecting, by at least one processor, one or more patterns on to a work surface via a projector; capturing, by the at least one processor, images of the projected one or more patterns via one or more cameras; performing, by the at least one processor, analysis on the captured images of the one or more patterns; determining, by the at least one processor, projector calibration parameters based at least in part on the performed analysis on the captured images; recalibrating, by the at least one processor, the projector system such that an image is projected onto the work surface by the projector at a correct position based on the determined projector calibration parameters; and using the projected, recalibrated image in a construction-related task.

39. A non-transitory computer-readable medium comprising a set of executable instructions, the set of executable instructions when executed by at least one processor causes the at least one processor to perform a method for recalibrating a projector system, the method comprising the steps of: projecting one or more patterns on to a work surface via a projector; capturing images of the projected one or more patterns via one or more cameras; performing analysis on the captured images of the one or more patterns; determining projector calibration parameters based at least in part on the performed analysis on the captured images; and recalibrating the projector system such that an image is projected onto the work surface by the projector at a correct position based on the determined projector calibration parameters.

40. A computer program product comprising a set of executable instructions, the set of executable instructions when executed by at least one processor causes the at least one processor to perform the method for recalibrating a projector system according to claim 38.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIGS. 1 and 2 illustrate projector systems in accordance with one or more principles of the present invention.

[0012] FIG. 3 illustrates a flow diagram in accordance with one or more principles of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

[0013] The present invention is directed to correctly and accurately projecting, using a projector system, a two-dimensional image (e.g., a construction-related blueprint) onto an uneven work surface, such as corrugated steel sheet, and automatically recalibrating the projector system (either by the projector system on-site itself or the user on-site) so that the pixels of the image actually appear where they are supposed to appear on the work surface. Surface profiling and projecting an image onto the profiled surface by a projector system are described in U.S. application Ser. No. 15/639,308, filed Jun. 30, 2017, the contents of which are incorporated herein by reference in its entirety.

[0014] In one embodiment of the present invention, a projector system may include a projector and one or more cameras, and the components of the whole projector system, including the cameras, may be calibrated and installed/stabilized, only once, in the off-site location (e.g., factory) where the projection system is produced and/or assembled. For instance, it may be easier to make mechanically stable the one or more cameras of the projector system than the projector itself. Thus, the positional relation between the one or more cameras is a known (and, importantly, a calibrated) parameter that should not change due to time or external stresses as the projector system is being operated in the field, such as a construction site.

[0015] During operation of the projector system, one or more patterns may be projected onto the work surface. It is not necessary to know the exact location and geometry of the work surface. The projected one or more patterns may include a specific shape (e.g., square, rectangle, triangle, etc.) or a combination of different types of shapes. As the one or more patterns are projected onto the work surface, the one or more cameras may capture each of the projected patterns.

[0016] All the captured images of the patterns are then used to determine projector calibration parameters. In one example, the correspondences between points from all cameras and the projector are computed. Thereafter, the unknown projector calibration parameters may be solved by equation system(s), which will be further described below. The computed projector calibration parameters may then be used for projecting a positionally correct image onto the work surface. Thus, the projection of the image at the correct position based on the projector calibration parameters relates at least to the recalibration of the projector system. The determination of the projector calibration parameters may be performed on-site and/or in real time so that time and costs associated with recalibrating the projector system are saved.

[0017] One of the numerous advantages of the present invention, which is further described below, is that the determination of the projector calibration parameters is based on known parameters (e.g., the positional relationship between the multiple cameras in relation to the projector) that will not change or easily change over time and/or external stresses, such as temperature or mechanical damage or stress. Moreover, the recalibration of the projector system can be performed on-site, in real-time, and/or simultaneously while the projector system is profiling the work surface.

[0018] The invention described herein may be implemented on and executed by one or more computing devices and/or one or more processors. For instance, the projector system may have computing capabilities, by way of example, one or more processors, central processing units (CPUs), etc. As will be further described below, the computing associated with determining projector calibration parameters according to aspect(s) of the present invention may be executed by computing hardware in the projector system itself. Alternatively, the processing may be performed by a separate portable computing device, such as a laptop, tablet computer, or any other suitable type of mobile computing device that can be operated by a user.

[0019] FIG. 1 illustrates an example projector system 110 in accordance with one or more principles of the present invention. As shown, the projector system 110 includes one or more processors 112, memory 114 (which includes instructions 116 and data 118), at least one projector 119, at least one camera 120, and at least one interface 121. The processor 112 may instruct the components of the projector system 110 to perform various tasks based on the processing of information and/or data that may have been previously stored or have been received, such as based on the instructions 116 and/or data 118 stored in memory 114. The processor(s) 112 may be a standard processor, such as a central processing unit (CPU), graphics processor, or may be a dedicated processor, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0020] The instructions 116 may be one or more sets of computer-executable instructions (e.g., software) that can be implemented by the processor 112. Data 118 may include various types of information (which can be retrieved, manipulated and/or stored by the processor 112), such as information to be projected by the projector 119, information captured by the one or more cameras 120, etc.

[0021] The projector 119 may be any object, apparatus, or device with a system of lenses that is used to project rays of light that form an image that is generated by a processor or computing device for projection. The one or more cameras 120 may be an optical instrument, apparatus, or device that is sued for recording or capturing images, which may be still photographs and/or sequences of images constituting a video or movie. While FIG. 1 illustrates a camera, it may be understood that any type of image capturing device may be used, such as a thermographic camera (infrared), etc.

[0022] Interface 121 may be any component that allows interfacing with an operator or user. For example, interface 121 may be a device, port, or a connection that allows a user to communicate with the projector system 110, including but not limited to a touch-sensitive screen, microphone, camera, and may also include one or more input/output ports, such as a universal serial bus (USB) drive, various card readers, etc. The interface 121 may also include hardware and/or equipment for surface profiling, such as one or more sensors, one or more range meters, etc.

[0023] Additionally, the projector system 110 may be configured to communicate with other computing devices via network 130. For example, the projector system 110 may communicate with other projector systems, mobile computing devices (e.g., laptops, tablet computers, smartphones). The network 130 may be any type of network, such as LAN, WAN, Wi-Fi, Bluetooth, etc.

[0024] Although the processing related to determining the projector calibration parameters are carried out by the one or more processors 112 of the projector system 110, it may be understood that the processing may be performed by external computing devices and/or hardware, such as a mobile computing device, that may be communicating with the projector system 110 via the network 130.

[0025] FIG. 2 illustrates the various components of a projector system 200 in accordance with one or more principles of the present invention. As shown, the projector system 200 may be configured in a similar manner to the projector system 110 of FIG. 1. For example, the projector system 200 may include at least two different cameras 202 and 204, which are arranged on opposing sides of a projector 206. The projector 206 may project an image, such as a construction-related blueprint for drilling various holes, onto a work surface 208, which in FIG. 2, may be a corrugated steel sheet. And for determining projector calibration parameters, the projector 206 may also be configured to project one or more patterns, as described above. The two cameras 202 and 204 may each capture images of the projection of each of the patterns onto the work surface 208, as also shown in FIG. 2.

[0026] In addition, FIG. 2 illustrates a pattern generator for generating and projecting the various patterns and a calibration process or routine for determining the projector calibration parameters (which will be further described below), which are depicted inside of the dashed box, as shown in FIG. 2. The dashed box represents that the above-described pattern generation and calibration process/routine for determining the projector calibration parameters may be performed or executed by one or more processors and/or one or more computing devices. For instance, the one or more processors may be the one or more processors 112 illustrated in FIG. 1. In that regard, the one or more processors 112 may be configured generate one or more specific patterns associated with the calibration process, as described above, and further the one or more processors 112 may then be configured to determine all requisite projector calibration parameters for recalibrating the projector system in order to accurately project images onto the work surface.

[0027] Moreover, as described above, the two cameras 202 and 204 may be arranged in the projector system 200 in a mechanically stable manner relative to each other and also relative to the projector 206. It is understood that mechanical stability may be provided in any fashion, such as by way of screws, fasteners, adhesive substance, etc. The overall effect, for example, of mechanical stability is that the cameras 202 and 204 will not move or change in position regardless of time, temperature, and mechanical stress, especially compared to the position and stability of the projector 206. In that way, the two cameras 202 and 204 may also be smaller in size than the projector 206 in order to achieve the above-described effect.

[0028] The calibration process/routine for acquiring, analyzing, and/or determining the various projector calibration parameters will now be described. According to an exemplary embodiment of the present invention, a set of projector calibration parameters may include intrinsic parameters and extrinsic parameters. For instance, the intrinsic parameters (for a linear projector model, e.g., pinhole model) may include:

[0029] (i) projector focal lengths in x-direction and y-direction (f.sub.x, f.sub.y)

[0030] (ii) principal point in x-direction and y-direction (t.sub.x, t.sub.y), and

[0031] (iii) skew coefficient between x-axis and y-axis (s).

Moreover, the extrinsic parameters may include: [0032] (i) translation vector between the projector and the one or more cameras (which may be known as a camera assembly) (t), and [0033] (ii) rotation matrix between projector and camera assembly (R).
In instances, where the camera-projector model is more complex, the set of projector calibration parameters may also include a set of distortion coefficients, etc. Not all projector calibration parameters are required to be estimated at the same time, as it may be understood that only a subset of the parameters can be derived at a time.

[0034] In at least that regard, the complete camera-projector calibration matrix including the intrinsic and extrinsic parameters may be represented as follows:

[00001]$C=K*E=K*\left[R|t\right].Math..Math.\mathrm{where}.Math.$$K=\left[\begin{array}{ccc}{f}_x& s& t_u\\ 0& f_y& t_v\\ 0& 0& 1\end{array}\right]$

is the intrinsic calibration parameter matrix. The variables R and t are the rotation matrix and translation vector between a camera or projector and a world coordinate system, respectively. The variables t.sub.u and t.sub.v denote the translation between optical axis and image sensor coordinate system. The variable C is a [34] calibration matrix.

[0035] Once the complete calibration matrix C is known, the intrinsic calibration matrix K, the rotation matrix R and the translation vector t may be determined by standard matrix decomposition or other types of suitable decomposition techniques.

[0036] With respect to a projected point on the work surface, the projected point may be represented by the variable k and has homogenous coordinates:

P.sub.k=(x.sub.ky.sub.kz.sub.k1).sup.T

[0037] For example, the projected point originates from the pixel of the projector image, which may be represented by:

P.sub.p=C.sub.pP.sub.k(1)

[0038] The same projected point appears on a camera image plane (e.g., for the first camera) represented by:

P.sub.c1=C.sub.c1P.sub.k(2)

[0039] The projected point appears on a different camera image plane (e.g., for the second camera) represented by:

P.sub.c2=C.sub.c2P.sub.k(3)

where P.sub.p, P.sub.c1, and P.sub.a are of the form P=(u v m).sup.T.

[0040] It may be understood that the two-dimensional image coordinates that are observable are represented by:

(u/w,v/w)=(u,v)

[0041] By way of example, writing out the above-described matrix equations (1) to (3) results in six different scalar equations:

u.sub.p(C.sub.p.sup.(3)P.sub.k)C.sub.p.sup.(1)P.sub.k=0

v.sub.p(C.sub.p.sup.(3)P.sub.k)C.sub.p.sup.(2)P.sub.k=0

u.sub.c1(C.sub.c1.sup.(3)P.sub.k)C.sub.c1.sup.(1)P.sub.k=0

v.sub.c1(C.sub.c1.sup.(3)P.sub.k)C.sub.c1.sup.(2)P.sub.k=0

u.sub.c2(C.sub.c2.sup.(3)P.sub.k)C.sub.c2.sup.(1)P.sub.k=0

v.sub.c2(C.sub.c2.sup.(3)P.sub.k)C.sub.c2.sup.(2)P.sub.k=0

where C.sup.(n) is the n-th row of the matrix C.

[0042] The camera calibration matrices C.sub.c1 and C.sub.c2 are known based on how the cameras were calibrated off-site (e.g., at the factory). The pixel coordinates of the projector u.sub.p, v.sub.p and the cameras u.sub.c1, v.sub.c1 and u.sub.c2, v.sub.c2 are known from the prior step of determining these correspondences. However, the locations P.sub.k of the points on the work surface are not known and the projector calibration matrix C.sub.p is not known.

[0043] In order to determine the locations P.sub.k of the points on the work surface and the projector calibration matrix C.sub.p, any solver for (e.g., nonlinear) overdetermined equation systems, such as the Levenberg-Marquardt algorithm, may be used to find a solution for the projector calibration matrix C.sub.p. Moreover, as a byproduct, the point locations P.sub.k can also be solved and a three-dimensional depth map of the work surface may be obtained. The determined projector calibration parameters and/or locations of the points may then be used to recalibrate the projector system, if needed.

[0044] It may be understood that in the case where only the correspondence of one point P.sub.1 is present, there is a system of equations with six equations and 15 unknowns. For every point that is added, six new equations are generated with three more new unknowns. As such, at least four points may be needed such that 24 equations for 24 unknowns are generated to obtain a unique solution to the equation system. Practically, many more points may be used, which makes the equation system overdetermined. Moreover, it may be understood that the above-described determination principles apply to a different number of cameras. And depending on the number of cameras, more or less correspondence points may be needed for the solution to become unique.

[0045] Variants and/or alternative embodiments of the above-described embodiment of the present invention will now be described.

[0046] In one embodiment, projector recalibration may be implemented and combined with the step of surface profiling (as further described U.S. application Ser. No. 15/639,308, as set forth above). In that regard, recalibration and surface profiling may be executed by the projector system simultaneously. Accordingly, for example, an alternative to projecting specific patterns is projecting other types of patterns (e.g., decoded patterns) used by a structured light approach to surface profiling may be used as input for solving the equation system(s) for the unknown projector calibration parameters.

[0047] In another embodiment, an alternative to projecting specific patterns onto the work surface to determine the projector calibration parameters is using the regular user image (e.g., construction-related blueprint image) that is projected onto the surface during operation on-site (for example, at a construction site).

[0048] In yet another embodiment, the change in projector calibration parameters due to temperature or mechanical stress may be relatively small in practical situations, which results in small amounts of parameter deviation from the factory calibration parameters. In order to account for these small deviations, the solution for the equation system above may be constrained to be close to a given set of calibration parameters, which can be derived, for instance, from the actual factory calibration.

[0049] In a further embodiment, a positioning system (which includes the one or more cameras) may separately be attached or snapped onto an off-the-shelf projector. If there is at least one camera with an overlapping field of view with the projector, the techniques described in the present invention may be used to determine the intrinsic (e.g., focal length, etc.) and extrinsic (e.g., relative position of projector to positioning system) projector calibration parameters. In at least that regard, the off-the-shelf projector may be used to project a positionally correct image on the work surface.

[0050] In yet a further embodiment, the present invention may be applied to any projector-camera system, not only in the visual light spectrum, but also for example in the infrared spectrum, as numerous commercial three-dimensional sensors use projector-camera pairs (such as infrared).

[0051] FIG. 3 illustrates a flow diagram 300 of recalibrating a projector system in accordance with one or more principles of the present invention. It may be understood that the steps of the flow diagram 300 may be performed or executed by one or more processors of a computing device, whether it may be via the example system 100 of FIG. 1 or via the one or more processors 112 of the projector system 110. Moreover, it may be understood that order of the steps in FIG. 3 may not be limited thereto, but may be arranged in any suitable order.

[0052] In step 302, a projector of the projector system may project on to a work surface one or more specific patterns. In step 304, images of each of the projected patterns may be captured by one or more cameras of the projector system. As described above, the specific patterns may include any suitable geometric shape, e.g., rectangles, squares, triangles, etc. or any suitable combination thereof.

[0053] In step 306, analysis is performed by the one or more processors on the captured images. The analysis may involve and include ascertaining various parameters from the projected patterns in order to solve the equation system(s), described above, along with the parameters that are already known, such as the positional relationship of the cameras with respect to each other and the projector, etc.

[0054] In step 308, the projector calibration parameters are determined based on the analysis in step 306, and in step 310, the projector system recalibrates in order to project an image onto the work surface at a correct position based on the determined projector calibration parameters.

[0055] Numerous advantageous of the present invention, include but are not limited to: the projector not having to be stable over time and temperature; recalibration of the projector may be performed in the field; recalibration may be done without user intervention; no specific calibration setup is required; recalibration and surface profiling may be performed simultaneously; and the projector focus may be adjusted manually to show a crisp image and the projector may automatically adjust to this focus setting so as to guarantee correct position and projection. An overarching advantage of the present invention is that recalibrating the projector system, for example on-site and in real-time, will increase overall accuracy of the projector system, especially in fields where precision and accuracy of the projection of an image is required. The recalibration is, thus, performed essentially in the hands of the user and/or operator, without having to send the projector off-site for recalibration.

[0056] The foregoing invention has been set forth merely to illustrate the invention and is not intended to be limiting. Since modifications of the disclosed embodiments incorporating the spirit and substance of the invention may occur to persons skilled in the art, the invention should be construed to include everything within the scope of the appended claims and equivalents thereof. Although the present disclosure uses terminology and acronyms that may not be familiar to the layperson, those skilled in the art will be familiar with the terminology and acronyms used herein.