Calibration method, calibration system, and program

Abstract

This method is for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft. The method includes: acquiring first captured image data based on first image data; acquiring second captured image data based on second image data different from the first image data; and calibrating the coordinate system of the image capture device and the coordinate system of the robot arm, using the first captured image data and the second captured image data.

Claims

1. A method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm comprising a drive shaft, the method comprising: causing the display device to display a first image based on first image data; capturing an image of the displayed first image and acquiring first captured image data, using the image capture device; causing the display device to display a second image based on second image data different from the first image data; capturing an image of the displayed second image and acquiring second captured image data, using the image capture device; and calibrating the coordinate system of the image capture device and the coordinate system of the robot arm using the first captured image data and the second captured image data.

2. The calibration method according to claim 1, wherein data indicating a pattern in an image to be displayed or data indicating a brightness of the image to be displayed differs between the first image data and the second image data.

3. The calibration method according to claim 1, wherein each of the first image data and the second image data comprise data indicating a pattern in an image to be displayed, and data regarding a color of at least one pixel in the image to be displayed differs between the first image data and the second image data.

4. The calibration method according to claim 1, further comprising: changing relative positions of the display device and the image capture device using the robot arm, after acquiring the first captured image data, wherein acquiring the second captured image data is performed after changing the relative positions.

5. The calibration method according to claim 1, wherein, while causing the display device to display the first image and causing the display device to display the second image, in the first image data or the second image data, a size of a pattern in the first image or the second image to be displayed is changed in accordance with coordinates of a leading end of the robot arm.

6. The calibration method according to claim 1, wherein, while causing the display device to display the first image and causing the display device to display the second image, in the first image data or the second image data, a size of a pattern in the first image or the second image to be displayed is changed in accordance with a distance between the image capture device and the display device.

7. The calibration method according to claim 1, further comprising: changing a relative angle between the display device and the image capture device using the robot arm, after acquiring the first captured image data, wherein at least data indicating a brightness of an image to be displayed differs between the first image data and the second image data.

8. The calibration method according to claim 1, wherein a relative positional orientation of the image capture device with respect to the display device while acquiring the first captured image data and a relative positional orientation of the image capture device with respect to the display device while acquiring the second captured image data are the same.

9. The calibration method according to claim 1, wherein the display device includes a sensor for measuring a brightness, and a processor configured with a program to perform operations comprising operation as a computing unit configured to change the first image data or the second image data based on a detection value from the sensor, and while causing the display device to display the first image and causing the display device to display the second image, in the first image data or the second image data, a brightness of the first image or the second image to be displayed is changed in accordance with the detection value from the sensor for measuring the brightness.

10. The calibration method according to claim 1, wherein the display device comprises a processor configured with a program to perform operations comprising: operation as an input unit configured to input information; and operation as a computing unit configured to change the first image data or the second image data based on the input information, and while causing the display device to display the first image and causing the display device to display the second image, in the first image data or the second image data, a size and a brightness of a pattern in the first image or the second image to be displayed are changed in accordance with the input information.

11. The calibration method according to claim 1, wherein the display device and the robot arm are fixed to each other, the display device includes a sensor for measuring an amount of movement, and in the first image data or the second image data, a size of a pattern in the first image or the second image to be displayed is determined based on the amount of movement measured by the sensor.

12. The calibration method according to claim 1, wherein the display device and the robot arm are fixed to each other, the display device includes a sensor for measuring an inclination, and data indicating a brightness of an image to be displayed based on at least one of the first image data and the second image data is determined based on a value of the inclination measured by the sensor.

13. A method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm comprising a drive shaft, the method comprising: causing the display device to display a first image based on first image data; capturing an image of the displayed first image and acquiring first captured image data, using the image capture device; causing the display device to display a second image based on second image data different from the first image data; capturing an image of the displayed second image and acquiring second captured image data, using the image capture device; and calibrating a relative positional relationship between the image capture device and the robot arm, using the first captured image data and the second captured image data.

14. The calibration method according to claim 13, wherein, in the first image data, data indicating a pattern in an image to be displayed or data indicating a brightness of the image to be displayed is determined in accordance with an image-capturing environment.

15. A non-transitory computer-readable storage recording medium storing a program for performing a method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm comprising a drive shaft, the program causing a computer to perform operations comprising: causing the display device to display a first image based on first image data; causing the image capture device to capture an image of the displayed first image to acquire first captured image data; causing the display device to display a second image based on second image data different from the first image data; causing the image capture device to capture an image of the displayed second image to acquire second captured image data; and calibrating the coordinate system of the image capture device and the coordinate system of the robot arm using the first captured image data and the second captured image data.

16. The non-transitory computer-readable recording medium storing the program according to claim 15, wherein data indicating a pattern in an image to be displayed or data indicating a brightness of the image to be displayed differs between the first image data and the second image data.

17. The non-transitory computer-readable recording medium storing the program according to claim 15, wherein each of the first image data and the second image data comprise data indicating a pattern in an image to be displayed, and data regarding a color of at least one pixel in the image to be displayed on the display device differs between the first image data and the second image data.

18. A calibration system comprising: a display device configured to display a first image based on first image data, and display a second image based on second image data different from the first image data; an image capture device configured to capture an image of each of the first image and the second image, and acquire first captured image data and second captured image data; a robot arm to which one of the display device and the image capture device is fixed, the robot arm comprising a drive shaft; and a processor configured with a program to perform operations comprising operation as a computing unit configured to perform computation to calibrate a coordinate system of the image capture device and a coordinate system of the robot arm, using the first captured image data and the second captured image data.

19. The calibration system according to claim 18, wherein data indicating a pattern in an image to be displayed or data indicating a brightness of the image to be displayed differs between the first image data and the second image data.

20. The calibration system according to claim 18, wherein each of the first image data and the second image data comprises data indicating a pattern in an image to be displayed, and data regarding a color of at least one pixel in the image to be displayed on the display device differs between the first image data and the second image data.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a functional block diagram illustrating a calibration system.

(2) FIG. 2 is a schematic diagram illustrating a calibration method in a case where a display 22 and a robot are fixed to each other.

(3) FIG. 3 is a diagram illustrating an initial setting procedure for a display device.

(4) FIGS. 4A to 4D are diagrams illustrating examples of calibration objects.

(5) FIG. 5 is a flowchart illustrating a calibration method.

(6) FIG. 6 is a schematic diagram illustrating a calibration method in a case where an image sensor and a robot are fixed to each other.

(7) FIG. 7 is a diagram illustrating a screen for inputting relative positional orientations of an image sensor and a robot.

DETAILED DESCRIPTION

(8) Hereinafter, embodiments will be described with reference to the attached drawings (note that, in the diagrams, portions assigned the same reference signs have the same or similar configurations). The calibration method that will be described below in the embodiments is for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed. A calibration pattern that is based on image data is displayed on the display device.

(9) In the calibration method described in any of the following embodiments, the display device has a display for displaying a plurality of image patterns in which at least one of the calibration pattern and the brightness of the image when the calibration pattern is displayed differs.

(10) In the calibration method described in any of the following embodiments, the display device is configured to change the image pattern and change the form of display for the image to be displayed on the display, in accordance with an image-capturing environment. Image-capturing environment refers to an environment that is due to a configuration of the robot system, and includes, for example, a relative distance between the image capture device and the display and relative orientations thereof, which serve as relationships between the display and the image capture device. Also, image-capturing environment refers to an environment around the robot system, and is, for example, the brightness of the surroundings when the image capture device captures an image of an image pattern displayed on the display. With this configuration, the calibration can be performed using an appropriate image pattern in accordance with a change in the image-capturing environment, and accordingly, the calibration accuracy can be improved.

(11) In the calibration method described in any of the following embodiments, the display device may determine the image pattern to be displayed on the display at a measurement point at which the image capture device captures an image of the display, in accordance with the image-capturing environment. With this configuration, the calibration can be performed using an appropriate image pattern in accordance with the image-capturing environment at the measurement point, and accordingly, the calibration accuracy can be improved.

(12) In the calibration method described in any of the following embodiments, the display device is configured to change the image pattern to be displayed on the display at a measurement point at which the image capture device captures an image of the display, and displays a plurality of images in different forms of display, on the display. With this configuration, the quantity of information regarding the image data acquired at a measurement point can be increased, and thus the calibration accuracy can be improved.

First Embodiment

(13) FIG. 1 is a functional block diagram of a calibration system 10 according to a first embodiment. This calibration system 10 has a robot R, and a robot controller 1 that has a robot control unit 12 for controlling the robot R. The calibration system 10 also includes an image sensor S, which serves as an image capture device. The robot controller 1 is configured to control operations of the robot R based on images captured by the image sensor S. In an embodiment, the system that includes the robot R, the robot controller 1, and the image sensor S is called a robot system. The calibration system 10 includes an image sensor controller 2 that has a sensor control unit 14 for controlling the image sensor S, an image processing unit 16 for processing images captured by the image sensor S, and a parameter calculation unit 18 (computing unit) for calculating parameters of a calibration between the robot R and the image sensor S. The calibration system 10 also includes a display device D that has a display 22 and a display control unit 20 for displaying images on this display 22. The control units, processing unit, and calculation unit are realized by a later-described hardware processor executing predetermined programs. The control units, processing unit, and calculation unit can exchange data with each other via wired or wireless communication.

(14) The robot R is a multi-axis articulated robot arm R, which includes a base fixed to a floor surface, a plurality of joints that function as movable shafts, and a plurality of links that rotate together with the joints. The links are movably connected to each other via corresponding joints. A link at a leading end of the robot arm R can be connected to an end effector E. In an embodiment, the robot arm R is a vertical articulated robot that has six degrees of freedom and in which the base and the plurality of links are connected in series via the joints.

(15) The robot control unit 12 is arranged in a robot controller that has a storage device that stores a control program for controlling the robot R, and a processor for outputting a control signal for driving a servomotor provided in each joint of the robot R to rotate the servomotor in accordance with this control program.

(16) The image sensor S is an image capture device for capturing an image to acquire captured image data, and is constituted by a CCD camera, for example.

(17) The sensor control unit 14 outputs a control signal for controlling the image sensor S. The image processing unit 16 performs image processing on the captured image data acquired by the image sensor S, and outputs the image-processed captured image data to the parameter calculation unit 18. The sensor control unit 14 has a storage device that stores a program for computing and outputting a control signal for controlling the image sensor S, and a processor for performing computing processing in accordance with the program. The image processing unit 16 has a storage device that stores a program for performing image processing on the captured image data and outputting the processing results to the parameter calculation unit 18, and a processor for performing computing processing in accordance with the program. Note that the sensor control unit 14 and the image processing unit 16 may be configured integrally and have a shared storage device and processor. The sensor control unit 14 and the image processing unit 16 may be configured integrally with the later-described parameter calculation unit 18 and have a shared storage device and processor.

(18) The parameter calculation unit 18 is constituted by a computer program for performing the calibration method according to an embodiment, a storage device that stores this computer program, data being processed, later-described image data, and so on, and a processor that performs various kinds of computing processing in accordance with the computer program.

(19) A processor that is used in common for computing processing of each of the robot control unit 12, the sensor control unit 14, the image processing unit 16, the parameter calculation unit 18, and the later-described display control unit 20, or computing processing of at least two units selected from thereamong may be a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), and an ASIC (Application Specific Integrated Circuit), together with a CPU or solely. A storage device that is used in common for computing processing of each of the robot control unit 12, the sensor control unit 14, the image processing unit 16, the parameter calculation unit 18, and the later-described display control unit 20, or computing processing of at least two units selected from thereamong has a nonvolatile storage device such as an HDD or an SSD, and a volatile storage device such as a DRAM or an SRAM.

(20) In an embodiment, the parameter calculation unit 18 is provided in the image sensor controller 2 (on the image sensor S side), and computing processing required for the calibration method according to an embodiment is performed by the processor provided in the parameter calculation unit 18. By thus using a processor other than the processor in the robot control unit 12, the load of the computing processing required for command processing for controlling the robot R and calculation of calibration parameters can be distributed.

(21) In another embodiment, at least some of the robot control unit 12, the image processing unit 16, and the parameter calculation unit 18 may be realized by using shared hardware. For example, a calibration may be performed using the processor in the robot controller, or the control program for controlling the robot R and the program for performing a calibration may be stored in a shared storage device. In this regard, at least two of the robot controller 1, the image sensor controller 2, and the display device D may be configured integrally and share the storage device for storing the programs and the processor for performing computing processing.

(22) The display control unit 20 generates image data for displaying various calibration patterns on the display 22, and outputs the generated image data to the display 22. The display 22 is constituted by, for example, a liquid-crystal display and so on, and displays an image based on the image data received from the display control unit 20. For example, a commercially-available tablet, smartphone, or the like can be used as hardware of the display device D constituted by the display control unit 20 and the display 22, but the display device D is not limited thereto and may alternatively be any other kind of liquid-crystal display device or organic EL display device. In an embodiment, the display control unit 20 and the display 22 are integrally expressed as the display device D. However, the display control unit 20 and the display 22 may be separately provided, rather than being integrated to serve as the display device D. In this case, the display 22 may include a storage device that stores a computer program for performing display processing or the like, data being processed, and so on, and a processor that performs various kinds of computing processing in accordance with the computer program.

(23) Note that, in an embodiment, the image data is stored in the storage device provided in the display control unit 20. The image data includes information regarding an image pattern that forms an image to be displayed on the display device D. The image pattern includes a calibration pattern, which is a pattern of an image to be displayed on the display device D, and information for determining the brightness of the image to be displayed on the display device D. The mode of the calibration pattern will be described later.

(24) FIG. 2 is a schematic diagram illustrating an example of performing a calibration with the display device D (display 22) fixed to the robot R. An image pattern can be displayed with various relative angles and relative positions of the display device D with respect to the image capture device S by moving the robot arm of the robot R while gripping the display device D. The display device D is gripped and fixed by the end effector E connected at the leading end of the robot R. Meanwhile, the image capture device S is fixed to a ceiling of a factory or the like, for example. Note that the method of fixing the display device D to the robot R is not limited to gripping using the end effector E. For example, the display device D may be fixed to the link at the leading end of the robot arm R via a connection structure such as a fixing jig. Alternatively, the display device D may be fixed to a link that constitutes the robot arm R via a predetermined connection structure. In the case of fixing the display device D to a link that constitutes the robot arm R, it is favorable that the display device D is fixed to the distal-most link of the robot arm R.

(25) FIG. 3 shows an example of an initial setting procedure for the display device D. Initially, the type of calibration pattern to be displayed on the display 22 is determined (S31). FIGS. 4A to 4D show examples of calibration patterns. FIG. 4A shows a dot image pattern constituted by a square frame line and a total of 49 (7 rows by 7 columns) black dots that are arranged at regular intervals in this frame line. Meanwhile, one of the corners is painted in a triangular shape to specify the direction of the square. That is to say, the calibration pattern shown in FIG. 4A is a pattern that includes black dots arranged at predetermined intervals in the frame line, and a polygonal mark marker for specifying the direction of the pattern. FIG. 4B shows image patterns of an AR marker (left) and a two-dimensional barcode (right). FIG. 4C shows an example of a checkerboard pattern (an image pattern in which black squares and white squares are arranged alternately), and FIG. 4D shows an example of an image pattern constituted by triangles. Various image patterns can be used in accordance with the positional relationship between the robot R and the image sensor S, the purpose of calibration, required accuracy, or the like. For example, the number and size of black dots in the aforementioned dot pattern can be set arbitrarily. In an embodiment, the dot image pattern is selected. The display control unit 20 is configured to output image data for displaying the selected image pattern to the display 22.

(26) Note that constituent elements of the calibration pattern included in each image pattern are pixels, and a pixel is the minimum unit of information regarding a color to be displayed on the display 22. If black and white of at least one pixel are reversed, image patterns are dealt with as different ones. For this reason, two image patterns that have similar shapes but have different sizes are different image patterns.

(27) Next, the size of the calibration pattern is determined (S32). In an embodiment, the size of the pattern is determined based on the relative distance between the display 22 and the image sensor S that is calculated as follows.

(28) First, the image sensor S is fixed to a ceiling or the like of a factory in which the robot R is installed. Accordingly, the position coordinates of the image sensor S are known. Similarly, the origin coordinates (base coordinates), which is obtained with the base of the robot R serving as a reference, are also known. For this reason, the relative positions of these coordinates and the relative distance therebetween are obtained, in advance, with an error of several centimeters by means of an installation design drawing or actual measurement, for example.

(29) Next, the leading end coordinates of the robot arm R are obtained. Specifically, the leading end coordinates are obtained by means of kinetics calculation based on the rotation angle of each movable shaft and length data (shape data) of each link, with the origin coordinates of the robot arm serving as a reference. Initially, the robot control unit 12 acquires information regarding the rotation angle of each movable shaft, and outputs the acquired information to the parameter calculation unit 18. The parameter calculation unit 18 calculates the leading end coordinates based on the received rotation angle information, as well as the length data of each link and the origin coordinates of the robot R that are known.

(30) Next, the coordinates of the display 22 (calibration object) are obtained. Specifically, the coordinates of the display 22 are obtained based on the known shape data (length data) of the display 22, with the leading end coordinates of the robot arm R serving as a reference. If the position and orientation of the display 22 (calibration object) relative to the leading end coordinate system of the robot arm R are changeable, the coordinates of the display 22 are obtained based not only on the shape data of the display 22, but also on a changed position and orientation. Next, the coordinates of the display 22 are obtained with the origin coordinates of the robot arm R serving as a reference, based on the coordinates of the display 22 relative to the leading end coordinate system of the robot arm R.

(31) Then, the relative distance between the display 22 and the image sensor S is calculated based on the calculated coordinates of the display 22 and position coordinates of the image sensor S.

(32) Note that the size of the later-described image pattern may be changed using the relative distance (second relative distance) between the leading end coordinates of the robot arm R and the image sensor S, in place of the relative distance (first relative distance) between the display 22 and the image sensor S. Since the display 22 (calibration object) is smaller than the robot arm R, and the positional orientation of the display 22 varies in accordance with operations of the robot arm R at the leading end coordinates, a change in the distance between the display 22 and the image sensor S can also be addressed even if the second relative distance is used in place of the first relative distance, and furthermore, the amount of computation can be reduced.

(33) The display control unit 20 generates image data in which the size of the image pattern selected in step S31 has been adjusted based on the relative distance calculated by the parameter calculation unit 18, and outputs the generated image data to the display 22. The display 22 displays a predetermined image pattern based on the received image data.

(34) Note that the method of calculating the relative distance between the display 22 and the image sensor S is not limited to the above-described method. For example, the relative distance may be calculated based on a change in the size of the pattern included in the calibration pattern image captured by the image sensor S. For example, an interval (first interval) between a first pattern element and a second pattern element that are predetermined and included in the calibration pattern is obtained in advance, and a relative distance (third relative distance) can be calculated based on information regarding the first interval, information indicating an interval (second interval) between the first pattern element and the second pattern element extracted from a captured image, and sensor parameters of the image sensor S. This configuration makes it possible to calculate the relative distance between the image sensor S and the display 22 based on the size of the calibration pattern in a captured image, the size changing in accordance with the distance between the display 22 and the image sensor S, and to vary the size the image pattern to be displayed, in accordance with the calculated relative distance.

(35) As described above, the size of the later-described image pattern can be varied in accordance with a change in the leading end coordinates of the robot arm by using any of the first relative distance, the second relative distance, and the third relative distance.

(36) Thus, in the calibration method according to an embodiment, the size of the image pattern to be displayed on the display 22 is determined in accordance with the relative distance between the display 22 and the image sensor S. In an embodiment, the size of the image pattern to be displayed (the area of a region enclosed by the square frame line) is predetermined so as to substantially occupy approximately half the image capture region of the image sensor S.

(37) Here, the size of an image pattern means the area of a portion enclosed by an outline of the image pattern, and can be expressed with the number of pixels in the display 22, for example. Note that the size of an image pattern may mean the size (area) of a pattern element that is present within the outline of the image pattern. For example, in the calibration pattern shown in FIG. 4A, the size of the image pattern may indicate the size of each of the black dots that are present within the frame line of the calibration pattern, and the size of the polygonal mark marker. Also, for example, in the calibration patterns shown in FIGS. 4B to 4D, the size of the image pattern may indicate the size of a pattern element that is present within the frame line of each of these calibration patterns. The size of a pattern element that is present within the frame line of each of these image patterns may be changed in accordance with a change in the size of the outline of the image pattern, or may be changed regardless of a change in the size of the outline of the image pattern.

(38) A specific calibration method will be described below using the drawings. In an embodiment, the image pattern displayed as a calibration pattern on the display 22 changes in accordance with various orientations that the robot arm R assumes.

(39) FIG. 5 is a flowchart of the calibration method according to an embodiment.

(40) Initially, the robot arm R assumes a predetermined orientation corresponding to a measurement point at which an image is to be captured, based on a control signal from the robot control unit 12 (S41). At this time, the orientation of the robot arm R is predetermined so that an entire display screen of the display device D is contained in an image capture region that can be captured by the image sensor S.

(41) Next, the relative distance between the leading end coordinates of the robot arm R assuming this orientation (the coordinates of the display 22) and the image sensor S is calculated using the above-described method by the parameter calculation unit 18, and the calculated relative distance is output to the display control unit 20. Subsequently, image data for displaying a dot pattern with a size that is based on the relative distance is generated by the display control unit 20, and is output to the display 22. Simultaneously, the sensor control unit 14 causes the image sensor S to capture an image of the image pattern displayed on the display 22, and acquires the captured image data (S42).

(42) Next, whether or not the number of times that an image is captured has reached a predetermined number of times (N times) is determined by the parameter calculation unit 18 (S43). If not, steps S42 and S43 are repeated. Specifically, processing to move to a measurement point different from the previous measurement point by causing the robot arm R to assume an orientation different from the previous one, processing to calculate the relative distance between the leading end coordinates (the coordinates of the display 22) at this time and the image sensor S, processing to generate image data for displaying a dot pattern with a corresponding size, and processing to cause the image sensor S to capture an image of the image pattern displayed on the display 22 to acquire the captured image data are repeatedly performed.

(43) Here, a smaller image pattern is displayed when the calculated relative distance is short than when the relative distance is long, and a larger image pattern is displayed when the relative distance is long than when the relative distance is short. For example, if the calculated relative distance is 1.2 times greater than that calculated at the previous measurement point, image data is generated so that each side of the square outer frame of the dot pattern and the diameter of each dot in the outer frame are 1.2 times greater than those at the previous time, and is displayed on the display 22. On the other hand, if the calculated relative distance is 0.8 times the relative distance calculated at the previous measurement point, image data is generated so that each side of the outer frame and the diameter of each dot in the outer frame are 0.8 times smaller than those at the previous time, and is displayed on the display 22. This configuration makes it possible to keep the size of the image pattern relative to the image capture region of the image sensor S to a predetermined value or more, and to thus improve the calibration accuracy.

(44) Note that not only dot image patterns with different sizes but also image pattern with different forms may be displayed, and images thereof may be captured. For example, the calibration accuracy can be improved by displaying image patterns in which black and white are reversed and acquiring captured image data for the respective images. That is to say, in at least two image patterns to be displayed, color information regarding at least one of the pixels corresponding to the display device D may be differentiated between a first image pattern and a second image pattern. More specifically, in the first image pattern and the second image pattern, white and black may be reversed in at least one of the pixels corresponding to the display device D. This configuration makes it possible to increase the quantity of acquirable information regarding captured images and improve the calibration accuracy.

(45) The size of the image pattern displayed on the display 22 may be continuously changed while moving the robot arm R to acquire captured image data of the image patterns of the respective sizes. For example, the image pattern displayed on the display 22 may be reduced (or increased) by a fixed size while controlling the robot arm R so that the display 22 approaches (or moves away from) the image sensor S at a fixed speed.

(46) If the number of times that an image is captured has reached the predetermined number of times (N times), and images of N image patterns displayed for N orientations have been captured, the positional orientation of the calibration object (the display 22 on which the calibration pattern is displayed) is calculated based on N sets of captured image data by the parameter calculation unit 18 (S44).

(47) Next, relative positional orientations of the robot and the sensor are obtained by the parameter calculation unit 18 based on the calculated positional orientation of the calibration object, and the positional orientation of the tool (S45). Specifically, a relative positional relationship between the robot and the sensor is acquired based on the following equation.
.sup.camH.sub.cal=.sup.camH.sub.base*.sup.baseH.sub.tool*.sup.toolH.sub.cal

(48) Here, .sup.camH.sub.cal in the left-hand side denotes a transformation matrix (including a rotation matrix and a translation vector) that indicates the positional orientation of the calibration object based on the coordinate system of the image capture device (camera), and is calculated based on the captured image data, as in the above-described steps.

(49) .sup.baseH.sub.tool denotes a transformation matrix that indicates the positional orientation of the leading end of the robot arm R that is based on the origin coordinate system thereof, and is calculated, as described above, by means of kinetics calculation based on the rotation angle of each movable shaft constituting the orientation of the robot arm R when the captured image data was acquired, and the known length data of each link.

(50) .sup.toolH.sub.cal denotes a transformation matrix indicating the positional orientation of the calibration object based on the leading end coordinate system of the robot arm R, and can be acquired, in advanced, based on the size of the end effector E and the display device D, for example. If the position and orientation of the calibration object relative to the leading end coordinate system of the robot arm R do not change during at least two times of image capturing, the computation of the transformation matrix expressed as .sup.toolH.sub.cal can be omitted. The cases where the computation of the transformation matrix expressed as .sup.toolH.sub.cal can be omitted include, for example, a case where image capturing is performed at least twice under a condition that the calibration object is fixed to the leading end of the robot arm R. In this case, when computation for a calibration is performed using two sets of captured image data acquired in association with each time of image capturing, a condition that the positional orientation of the calibration object relative to the leading end coordinate system of the robot arm R does not change can be used. Accordingly, the computation of the transformation matrix expressed as .sup.toolH.sub.cal can be omitted.

(51) Accordingly, a transformation matrix .sup.camH.sub.base that indicates the positional orientation of the robot R at the origin coordinates seen from the coordinate system of the image sensor S, or an inverse matrix .sup.baseH.sub.cam of this transformation matrix can be acquired as information indicating a relative positional relationship between the robot R and the image sensor S.

(52) Through the above steps, a calibration is complete in which the relative positional relationship between the image sensor S, which is an image capture device, and the robot arm R, which is a robot, is acquired.

(53) Modifications

(54) Modifications of the above-described first embodiment will be described below. Redundant descriptions will be omitted, and different points will be described.

(55) In a first embodiment, the robot arm R and the display device D are fixed to each other, and captured image data required for a calibration is acquired while moving the display device D using the robot arm R. This modification will describe a method in which the robot arm R and the image sensor S are fixed to each other, and captured image data required for a calibration is acquired while moving the image sensor S using the robot arm R.

(56) First, the image sensor S is fixed to the robot arm R. Some commercially-available robot arms R are originally provided with an image sensor S. Such a robot arm R can perform a calibration using the image sensor S thereof. For a robot arm R that is not originally provided with the image sensor S, the image sensor S needs to be fixed to a link at the leading end, for example, using a jig or the like. Meanwhile, the display 22 is fixed at a predetermined position.

(57) FIG. 6 is a schematic diagram illustrating a state where an image of an image pattern displayed on the display 22 is captured using the robot arm R with the image sensor S fixed thereto.

(58) A specific calibration method will be described below using the drawings. In this modification as well, the calibration pattern included in the image pattern displayed on the display 22 changes in accordance with various orientations that the robot arm R assumes. The flowchart in FIG. 5 can be used as that of the calibration method according to this modification. The robot arm R assumes a predetermined orientation so that an entire display screen of the display 22 is contained in an image capture region that can be captured by the image sensor S (S41).

(59) Next, the sensor control unit 14 causes the image sensor S to capture an image of the image pattern displayed on the display 22, and acquires captured image data (S42). However, the size of the image pattern can be determined using a method other than that in a first embodiment. For example, the size of the image pattern can be determined based on the relative distance between the leading end coordinates of the robot arm R and the position coordinates of the fixed display 22.

(60) Next, whether or not the number of times that an image is captured has reached a predetermined number of times (N times) is determined by the parameter calculation unit 18 (S43). If not, steps S42 and S43 are repeated. Specifically, the image sensor S captures images of various image patterns displayed on the display 22 to acquire the captured image data while the robot arm R is assuming various orientations.

(61) If the number of times that an image is captured has reached a predetermined number of times (N times), the positional orientation of the calibration object (the display 22 on which the calibration pattern is displayed) is calculated based on the captured image data by the parameter calculation unit 18 (S44).

(62) Then, the relative positional orientations of the robot and the sensor are obtained by the parameter calculation unit 18 based on the calculated positional orientation of the calibration object and the positional orientation of the tool (S45). Specifically, a relative positional relationship between the robot and the sensor is acquired based on the following equation.
.sup.camH.sub.cal=.sup.camH.sub.tool*.sup.toolH.sub.base*.sup.baseH.sub.cal

(63) Here, .sup.camH.sub.cal in the left-hand side denotes a transformation matrix (including a rotation matrix and a translation vector) that indicates the positional orientation of the calibration object based on the coordinate system of the camera, and is calculated based on the captured image data, as in the above-described steps.

(64) .sup.toolH.sub.base denotes an inverse matrix of the transformation matrix that indicates the positional orientation at the leading end that is based on the origin coordinate system of the robot arm R in a first embodiment, and is calculated by means of kinetics calculation based on the rotation angle of each movable shaft constituting the orientation of the robot arm R when the captured image data was acquired, and the known length data of each link.

(65) .sup.baseH.sub.cal a transformation matrix that indicates the positional orientation of the calibration object that is based on the origin coordinate system of the robot arm R, and can be acquired in advance, based on the position coordinates of the display device 22, the size of the display device D, and the like. If the position and orientation of the calibration object relative to the origin coordinate system of the robot arm R do not change during at least two times of image capturing, the computation of the transformation matrix expressed as .sup.baseH.sub.cal can be omitted. The cases where the computation of the transformation matrix expressed as .sup.baseH.sub.cal can be omitted include, for example, a case where the robot arm R is operated, images of the calibration object can be captured using the image sensor S before and after the orientation of the robot arm R is changed, and the location to which the calibration object is fixed does not need to be changed.

(66) Accordingly, a transformation matrix .sup.camH.sub.tool that indicates the positional orientation of the robot R at the leading end coordinates seen from the coordinate system of the image sensor S, or an inverse matrix .sup.toolH.sub.cam of this transformation matrix can be acquired as information indicating a relative positional relationship between the robot R and the image sensor S.

(67) Through the above steps, a calibration is complete in which the relative positional relationship between the image sensor S, which is an image capture device, and the leading end coordinates of the robot arm R is acquired.

(68) As described above, with the calibration method based on an embodiment and the modification thereof, a relative positional relationship between the image sensor S, which is an image capture device, and the robot arm R can be acquired without preparing calibration objects with a plurality of patterns printed thereon.

(69) Note that the size of the square frame of the dot image pattern, the number of dots therein, and the interval between dots can be set arbitrarily. Various image patterns may be combined to be used in a calibration in accordance with the purpose of calibration, or the like.

(70) The calibration system 10 may also include an output unit for outputting a relative positional relationship between the robot R and the image sensor S.

(71) The size of the image pattern may not be determined based on the relative distance between the display 22 and the leading end coordinates of the robot arm R, and may alternatively be determined based on other indexes. For example, the size of the image pattern may be determined in accordance with the distance between a predetermined point and the leading end coordinates of the robot R.

(72) Leading end coordinates is intended to include not only the coordinates of the link located at the tip, of the links constituting the robot arm R, but also the coordinates of a point that is located forward of the tip of (i.e. located at a terminal of) the robot arm, such as the leading end or intermediate coordinates of a tool (end effector).

Second Embodiment

(73) A second embodiment will describe points different from a first embodiment, while omitting descriptions of the same features.

(74) In a first embodiment, the size of the image pattern is changed based on the relative distance between the image sensor S and the leading end of the robot arm R. In an embodiment, the size of the calibration pattern (image pattern) is changed based on the amount of blur in the calibration pattern whose image is captured by the image sensor S.

(75) First, a method of estimating the distance based on the amount of blur will be described. If the focal length of an imaging plane lens of the image sensor S is denoted as f, and the distance between the image sensor S and a light source A is denoted as u, the distance v at which the light source A is brought into focus satisfies a relationship expressed as f.sup.1=u.sup.1+v.sup.1. Accordingly, if the distance r from the imaging plane lens to the imaging plane satisfies r=v, an in-focus image can be obtained. On the other hand, if rv, when the aperture of the image sensor S (camera) is denoted as p, the light source A is projected as a circle having a diameter q=p*|vr|/v on the imaging plane, and a blur with an amount of blur q occurs.

(76) Accordingly, the amount of blur q can be obtained by preparing, as an image pattern, image data that includes a point, capturing an image of the image pattern, and performing image recognition on captured image data to check the diameter of the point appearing as a circle in the captured image. Then, the distance u between the image sensor S and the calibration object can be estimated based on p, r, and f, which are known, and the above equation.

(77) By changing the size of the image pattern based on the distance u, the size of the calibration pattern to be displayed can be increased even if the image sensor S and the display 22 move away from each other. Also, even if the image sensor S and the display 22 move closer to each other, the entire calibration pattern can be displayed within the image capture region of the image sensor S by reducing the size of the calibration pattern to be displayed.

(78) Furthermore, since computing processing to calculate the leading end coordinates of the robot R and so on can be omitted, the load on the processors can be reduced.

(79) Note that, instead of the distance estimation based on the amount of blur, a configuration may alternatively be employed in which a marker is displayed as an image pattern displayed on the image sensor, and the distance between a plurality of feature points in the marker is calculated by means of image recognition, and the size of the image pattern may be changed based on this distance (or the amount of change in the distance between the feature points according to a change in the orientation).

Third Embodiment

(80) (Halation) A third embodiment will describe points different from the above-described embodiments, while omitting the same features.

(81) A calibration system according to an embodiment includes an illuminance sensor. The brightness of the image pattern displayed on the display 22 is differentiated in accordance with the value of the ambient illuminance obtained by the illuminance sensor. It is favorable that the illuminance sensor is integrally provided with the display 22. Specifically, in the case of high illuminance (e.g. in the case where a display surface of the display 22 opposes lighting on the ceiling, and thus the light reflected off the display surface is received by the image sensor 5), the display 22 is caused to display an image pattern based on image data with which the image pattern is relatively darkly displayed compared with the case of low illuminance. In the case of low illuminance (e.g. in the case where the display surface of the display 22 faces toward a floor, and thus the lighting on the ceiling is hardly received by the image sensor 5), the display 22 is caused to display an image pattern based on image data with which the image pattern is relatively brightly displayed compared with the case of high illuminance. It is thereby possible to perform a calibration while reducing the influence of a halation (a phenomenon in which the periphery of a subject is blurred whitely and unclear due to an excessively strong light beam) that is caused due to, for example, the lighting installed on the ceiling.

(82) For example, degradation of the calibration accuracy due to a halation can be suppressed without changing the relative distance between the image sensor S and the display 22, by controlling the robot arm R so as to change the relative angle of the display 22 with respect to the image sensor S and changing the brightness of the displayed image in accordance with the output of an illuminometer.

(83) If, for example, the position of the lighting is known, a configuration may be employed in which image data that indicates the brightness is adjusted based on the positional orientation of the robot arm R, an estimated angle of the display surface of the display 22 relative to the floor surface, or the like.

(84) If a tablet-type mobile information processing terminal or a smartphone is used as the display device D, an illuminance sensor that is originally provided in this terminal may be used.

(85) In an embodiment, image data is intended to include not only the shape of a displayed image but also an item whose brightness is to be differentiated, and includes not only data indicating a brightness that is designated for each pixel, but also data indicating control data that defines the output of a backlight of the display 22, and the degree of brightness.

Fourth Embodiment

(86) (Use of functions of display device, input unit, sensor etc.) A fourth embodiment will describe points different from the above-described embodiments, while omitting the same features.

(87) In the above-described embodiments, the processors are used to calculate the relative distance between the image sensor S and the leading end of the robot arm R, and so on. However, by providing the display device D with various functions, the display device D can alternatively be used in complicated computing processing to be performed using processors or the like.

(88) For example, as shown in FIG. 7, a configuration can be employed in which an input I/F (input means) is provided in the display device D, and the input of estimated values of relative positional orientations of the robot R and the image sensor S, and a measurement range and visual field range of the image sensor S (or the input of a model number or the like of the image sensor S that defines these values) are received. In this configuration, an image pattern with an appropriate size, shape, and brightness can be automatically displayed in accordance with the input values. A configuration may also be employed in which the shapes of a plurality of calibration objects are selectably displayed in a scrollable manner on the display 22. Note that the mode of the input I/F is not particularly limited to the mode shown in FIG. 7. For example, information that determines the content of an image pattern may be input via a communication I/F that is provided in the display device D for wired or wireless electrical communication with an external device.

(89) A configuration may also be employed in which an acceleration sensor, a speed sensor, or a tilt (gyroscope) sensor is provided in the display device D or the robot arm R, and an image pattern with an appropriate size, shape, and brightness is automatically displayed based on measurement data from the sensor indicating the amount by which the robot arm R has moved or the inclination of the robot arm R.

(90) This configuration makes it possible to objectively select an image pattern with an appropriate shape, size, or the like. In addition, a physical configuration or the like for transmission and reception, i.e. for receiving data for calculating the relative distance or the like from the robot arm R is not needed, and image data for displaying an image pattern can be determined only with the display device D. Furthermore, in the case of using a mobile terminal (such as a tablet or a smartphone) as the display device D, an input means and various sensors that are originally provided therein can be used.

(91) Note that all of the embodiments may employ a configuration in which images are displayed based on different image patterns at the same measurement point, and computation for a calibration is performed based on a plurality of captured images acquired at the same measurement point. Here, it is favorable that different image patterns are image patterns in which color information regarding at least one pixel corresponding to the display device D differs. It is particularly favorable that different image patterns are image patterns in which white and black are reversed in at least one pixel in the display device. This configuration makes it possible to increase the quantity of information regarding the captured images acquired at each measurement point, and improve the calibration accuracy. That is to say, value information on which calibration computation is based can be increased by displaying a plurality of types of image patterns in different modes at the same measurement point. Accordingly, with this configuration, computing processing for a calibration can be performed using a plurality of types of image patterns acquired at the same measurement point, and thus the calibration accuracy can be improved.

(92) The above-described embodiments are for facilitate understanding of the present invention, and is not for interpreting the present invention in a limited manner. The elements provided in the embodiments, the arrangement, material, conditions, shape, size, and the like of those elements are not limited to those described as an example, and may be changed as appropriate. Configurations described in different embodiments may be partially replaced, or may be combined.

(93) Note that, in this specification, unit, means, and procedure do not simply mean physical configurations, but also include the cases where processing performed by such unit and the like is realized by software. Processing to be performed by one unit or the like or devices may be performed by two or more physical configurations or devices. Processing to be performed by two or more units or the like or devices may be performed by one physical means or device.

(94) The above embodiments may also be described, entirely or partially, as in the following Notes, but are not limited thereto.

(95) (Note 1)

(96) A method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft, the method including:

(97) a step of causing the display device to display a first image based on first image data;

(98) a step of capturing an image of the first image displayed on the display device and acquiring first captured image data, using the image capture device;

(99) a step of causing the display device to display a second image based on second image data different from the first image data;

(100) a step of capturing an image of the second image displayed on the display device and acquiring second captured image data, using the image capture device; and

(101) a step of calibrating the coordinate system of the image capture device and the coordinate system of the robot arm using the first captured image data and the second captured image data.

(102) (Note 2)

(103) A method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft, the method including:

(104) a step of causing the display device to display a first image based on first image data;

(105) a step of capturing an image of the first image displayed on the display device and acquiring first captured image data, using the image capture device; and

(106) a step of calibrating the coordinate system of the image capture device and the coordinate system of the robot arm, using the first captured image data.

(107) (Note 3)

(108) A program for performing a method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft,

(109) the program for causing a computer to perform:

(110) causing the display device to display a first image based on first image data;

(111) causing the image capture device to capture an image of the first image displayed on the display device to acquire first captured image data;

(112) causing the display device to display a second image based on second image data different from the first image data;

(113) causing the image capture device to capture an image of the second image displayed on the display device to acquire second captured image data; and

(114) calibrating the coordinate system of the image capture device and the coordinate system of the robot arm using the first captured image data and the second captured image data.

(115) (Note 4)

(116) A calibration system comprising:

(117) a display device configured to display a first image based on first image data, and display a second image based on second image data different from the first image data;

(118) an image capture device configured to capture an image of each of the first image and the second image, and acquire first captured image data and second captured image data;

(119) a robot arm configured to change relative positions of the display device and the image capture device, one of the display device and the image capture device being fixed to the robot arm; and

(120) a computing unit configured to perform computation to calibrate a coordinate system of the image capture device and a coordinate system of the robot arm, using the first captured image data and the second captured image data.