ROBOT TRAJECTORY LEARNING BY DEMONSTRATION WITH PROBE SENSOR

Abstract

A robot learning system for trajectory learning of a robot (RB) having a robot arm between a base and a tool center point (TCP). A user interface allows the user to control the robot arm in order to follow a desired trajectory during a real-time. A probe sensor (PS) is mounted on the TCP during the learning session. The probe sensor (PS) measures a distance parameter (Z) indicative of distance from the TCP and a surface forming the trajectory to be followed, and an orientation parameter (X, Y) indicative of orientation of the TCP and the surface forming the trajectory to be followed. These distance and orientation data are provided as a feedback to the controller of the robot (CTL) during the real-time learning session, thereby allowing the robot controller software to assist the user in following a desired trajectory in a continuous manner. Especially, the probe sensor (PS) may have a displaceable tip (TP) to follow a surface and having a neutral or center position, and where the robot controller software controls the robot movements to seek the neutral or center position irrespective of the user's control inputs. Data (DT) is logged during the learning session, so as to allow later control of the robot (RB) in response to the data (DT) logged during the learning session.

Claims

1. A robot learning system for trajectory learning of an associated robot comprising a robot arm between a base and a tool center point, by demonstration from a user, the system comprising: a user interface configured for connection to a controller of the robot, so as to allow the user to control the robot arm in order to follow a desired trajectory during a real-time learning session, wherein the user interface comprises at least a first control element for being operated by the user, and being configured to control position in space of the tool center point of the robot, a probe sensor configured to be mounted on the tool center point during the real-time learning session, wherein the probe sensor is configured to measure a distance parameter indicative of distance from the tool center point and a surface forming the trajectory to be followed and an orientation parameter indicative of orientation of the tool center point and the surface forming the trajectory to be followed, and wherein the probe sensor (PS) is configured to continuously generate one or more signals corresponding to said distance and orientation parameters, and wherein said one or more signals is provided as a feedback to the controller of the robot during the real-time learning session, and a processor configured to log data in response to the user's operation of the at least first control element during the real-time learning session, or continuously logging data at a predetermined sample rate, so as to allow later control of the robot in response to the data logged during the learning session.

2-15. (canceled)

16. The robot learning system according to claim 1, wherein the probe sensor comprises a longitudinally displaceable rod connected to the tip at one end and connected to a base of the probe sensor at the opposite end, so as to allow sensing of a distance between the base of the probe sensor and the tip by a distance sensor.

17. The robot learning system according to claim 16, wherein the longitudinally displaceable rod and the base of the probe sensor are connected at a joint, so as to allow multidirectional movement of the longitudinally displaceable rod in relation to the base of the probe sensor, wherein a first angle sensor is configured to sense an angle between the base of the probe sensor and the displaceable rod.

18. The robot learning system according to claim 17, wherein the first angle sensor is configured to sense an angle in a first direction between the base of the probe sensor and the longitudinally displaceable rod, and wherein a second angle sensor is configured to sense an angle in a second direction between the base of the probe sensor and the longitudinally displaceable rod, wherein said first and second directions are different.

19. The robot learning system according to claim 1, wherein the probe sensor has a neutral or center position of its tip relative to its base.

20. The robot learning system according to claim 19, wherein the tip of the probe sensor is resiliently connected to its base, so that that the tip will return to the neutral or center position after being forced away from the neutral or center position.

21. The robot learning system according to claim 1, wherein the processor or robot controller is programmed to continuously calculate a transformation of the robot coordinates in response to data representing said signal from the probe sensor during the real-time learning session.

22. The robot learning system according to claim 21, wherein the processor or robot controller is programmed to control the robot in response to a combination of input from the user interface and data representing the signal from the probe sensor, during the real-time learning session.

23. The robot learning system according to claim 22, wherein the robot controller is programmed to move the robot in response to feedback from the probe sensor, during the real-time learning session, so as to minimize a deviation between an actual position of the tip of the probe sensor and a neutral or center position of the tip of the probe sensor.

24. The robot learning system according to claim 22, wherein the robot controller is programmed to move the robot so as to help the user in continuously controlling the robot to ensure that the tip of the probe sensor is in contact with a surface of an object to be followed.

25. The robot learning system according to claim 1, wherein the user interface comprises a first control element comprising a first joystick configured for operation by the user's one hand for control of position in space of the tool center point of the robot, wherein the user interface comprises a second control element comprising a second joystick for tilting or rotating the tool center point of the robot, wherein the second joystick is configured for simultaneous operation by the user's second hand.

26. The robot learning system according to claim 1, wherein the processor is configured to control the robot in response to the data logged during the learning session, and wherein the processor is programmed to calculate a transformation of the robot coordinates in response to an input regarding physical properties and further in response to known properties of the probe sensor.

27. A robot system comprising a robot comprising a robot arm with a plurality of moveable arm elements arranged between a base and a tool center point, wherein the tool center point is configured for mounting of a tool a robot controller configured to control movement of the robot, and a robot learning system according to claim 1.

28. A method for controlling a robot during trajectory learning of the robot by demonstration from a user to make the robot follow a desired trajectory during a real-time learning session, the method comprising: receiving distance input indicative of a distance from the tool center point (TCP) of the robot and a surface forming the trajectory to be followed during the real-time learning session, receiving orientation input indicative of an orientation of the tool center point relative to the surface forming the trajectory to be followed during the real-time learning session continuously controlling the robot in response to the user input, the distance input and the orientation input during the real-time learning session, and logging data in response to the user's operation of the at least first control element during the learning session so as to allow later control of the robot in response to the data logged during the learning session.

29. A computer program product having instructions which, when executed, cause a computing device or system comprising a processor to perform the method according to claim 28.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0049] The invention will now be described in more detail with regard to the accompanying figures of which

[0050] FIGS. 1a and 1b illustrate an overview of a system embodiment during a learning session (1a), and in an execution session (1b) where the robot performs a pre-recorded task,

[0051] FIGS. 2a and 2b illustrate the concept of independently controlling orientation or rotation (2a) and position (2b) of the robot with two joysticks,

[0052] FIG. 3 illustrates movements of a three-axis joystick, as an example of a control element,

[0053] FIG. 4 illustrates an example of a 3D probe sensor for placing at the TCP during the learning session,

[0054] FIG. 5 illustrates an example of a flow chart for a robot control loop state diagram during learning (to the left), and during execution (to the right),

[0055] FIGS. 6a, 6b, and 6c show photos of an implementation with two three-axis joysticks mounted on a six axes CRS A465 robot, and FIGS. 6a and 6c further show a probe sensor mounted on the TCP of the robot,

[0056] FIG. 7 illustrates steps of a method embodiment,

[0057] FIG. 8 illustrates preferred processing of sensor data from the probe sensor,

[0058] FIG. 9 illustrates a preferred algorithm for providing position feedback from the probe sensor for controlling the robot, and

[0059] FIG. 10 illustrates a photo of a probe sensor embodiment with a ball shaped tip and a protective bellow.

[0060] The figures illustrate specific ways of implementing the present invention and are not to be construed as being limiting to other possible embodiments falling within the scope of the attached claim set.

DETAILED DESCRIPTION OF THE INVENTION

[0061] FIG. 1a shows basic parts of a robot learning system system embodiment during a real-time trajectory learning session, while FIG. 1b shows the same robot RB in an execution process, e.g. a welding process, based on data DT logged during the learning session.

[0062] The robot RB has in the example an arm formed by 5 joint arm elements between a base and a tool center point TCP. A controller of the robot CTL serves to control actuators for moving the arm elements. A probe sensor PS with a probe tip PT is mounted on the TCP of the robot RB, so as to continuously sense distance and orientation between the probe tip PT and the TCP during the real-time trajectory learning session. This allows contact with the probe tip during the trajectory following on an object, e.g. a welding trace. A signal PTL_IN is generated by the probe sensor in response to the sensed distance and orientation.

[0063] The user interface comprises two three-axis joysticks J1, J2 both mounted on the robot arm, namely on the TPC, and on an elbow between two arm elements, a suitable distance (e.g. 30-80 cm) away from each other to allow the user to comfortably operate both joysticks J1, J2 simultaneously. A specific example of a low cost joystick is Apem 3140SAL600, which is based on Hall effect sensing technology.

[0064] In FIG. 1a, a human user (not shown) operates the two joysticks J1, J2 simultaneously, by respective hands, to cause the robot RB to move the position and orientation of its TCP so that the probe tip PT follows contact with a desired trajectory on an object. The splitting of the position and orientation tasks between the user's respective hands provides, especially in combination with the mounting points on the robot, provides an intuitive user interface. Preferably, the probe sensor PS is arranged such that the probe tip PT is in flexible connection with the TCP, thus allowing even easier following of a complicated shaped trajectory. The joystick J1 at the TCP controls position of the TCP, thus also position of the probe tip PT, while the other joystick J2 at the elbow controls orientation, i.e. rotation of the TCP, and thus also orientation or tilting of the probe sensor PS. This allows the user with an intuitive control of the robot movements during following of e.g. a welding trace in real-time. The two joysticks J1, J2 are arranged to generate respective electrical input signals P_IN, R_IN to a controller CTL of the robot RB, which generates control signals C_S to control actuators in robot RB accordingly. Further, an electrical signal PTL_IN from the probe sensor PS is also applied to the controller CTL of the robot with the purpose of logging data indicative of at least a distance between the probe tip PT and the TCP.

[0065] In the shown embodiment in FIGS. 1a and 1b, signals P_IN, R_IN, PTL_IN from the joysticks J1, J2 and the probe sensor PS are applied to the processor in the controller CTL of the robot RB, while control data DT are stored for later use, i.e. an indirect way of storing the signals P_IN, R_IN, PTL_IN from the joysticks J1, J2 and the probe sensor PS. However, these signals P_IN, R_IN, PTL_IN may alternatively be directly logged by a separate processor, e.g. a laptop computer. However, in both cases, the purpose is to sample the signals P_IN, R_IN, PTL_IN with a suitable time and magnitude resolution, and to preferably store such data DT in a digital format at a storage medium STM for later use, so as to allow later control of the robot RB in response to the data DT logged during the learning session.

[0066] In FIG. 1b, data DT stored on a storage medium STM during the real-time trajectory learning are played back to the controller CTL of the robot RB which now has a process tool PTL mounted on its TCP. The joysticks and now not in use, since the robot RB is now controlled by the stored data DT so as to cause the process tool PTL to follow the learned trajectory with respect to both position in space and orientation, i.e. tilting.

[0067] In the system, key elements are two three axes joysticks J1, J2 mounted in this example on a six axes industrial robot body, and a preferably three axes probe sensor PS. In the shown example the position controlling joystick J1 is mounted near to the 6'th joint of the robot, i.e. near the TCP. The orientation controlling joystick J2 is mounted near to the robot elbow, or at distance of human shoulder width, e.g. around 45-80 centimeters, in case of a large industrial robot. The three axes probe sensor PS is mounted on the TCP.

[0068] FIGS. 2a and 2b show the intuitive learning method by means of the two joysticks J1, J2 mounted on the robot as in FIGS. 1a, 1b, i.e. the relation between the joystick J1, J2 axes and the robot Cartesian axes are shown. The probe sensor tip is shown in contact with the surface of a work piece WP.

[0069] In the three sketches of FIG. 2a the orientation controlling by means of joystick J2 is illustrated, where the arrows on the sketches to the right indicate the preferred effect of movement of the orientation joystick J2 and its orientation effect on the robot, where the x and y movable axes of the joystick J2 controls the x or y rotation of the TCP. By rotating the joystick J2 handle the z axis rotation can be controlled by the user.

[0070] In FIG. 2b, the arrows on the sketches to the right indicate the preferred effect of movement of the position joystick J1 and its position effect on the robot TCP, where the x and y movable axes of the joystick J1 controls the x or y position of the TCP. By rotating the joystick J1 handle the z axis position can be controlled by the user.

[0071] FIG. 3 illustrates operation of a preferred control element, namely a three-axis joystick, seen from above. The x and y axes controls the x or y position or x or y rotation of the robot, while the z axis position or orientation is controlled by rotation the joystick handle. In an alternative implementation, the z axis may be controlled by moving the joystick handle up and down. Preferably, the joysticks are designed to enter a neutral center position when no force is applied to it, and preferably the robot is designed to stop moving, when this neutral center position is sensed.

[0072] FIG. 4 shows a sketch of a probe sensor PS embodiment. Preferably, the length of the probe sensor PS should be designed to be similar, or at least approximately, to the desired process tool to be used on the robot in the working situation. The design is based on a two axes 2A_S joystick sensor with a linear movably or displaceable probe rod P_R mounted on it, and with a sensor (not visible) capable of sensing the linear displacement of the probe rod P_R, e.g. a linear potentiometer, thereby generating a signal indicative of length Z. Angle potentiometers (not visible) may be use to generate signals indicative of orientation, since these potentiometers can sense angles in two perpendicular directions X and Y, and thus reflect the tilting of the probe rod around the joint where it is connected to the base of the probe sensor.

[0073] All the axes preferably have spring designs, so the axis will center the position when no forces are applied on the probe sensor PS. The probe tip PT is preferably designed for the actual workpiece and therefore must be easily replaceable. E.g. the probe tip PT may be formed as a small ball to be able to easily slide on a surface.

[0074] Equation (1) describes the transformation defined in h{1, . . . , n} where n is the number of records. The transformation in equation is calculated by using the rotation and position part in a homogeneous coordinate description. Equation (2) describes the transformation from the robot base to the end of the last joint for the i'th transformation. This equation is input to the robot controller for moving the robot around the scene. Equation (3) describes the transformation from the robot base to the end of the probe sensor for the i'th transformation. During the learning process, the two transformations on the right side are recorded for calculating the desired path during execution described in equation (4):

.sub.iT.sub.6Joint.sup.Probe(1)

.sub.iT.sub.Base.sup.6Joint(2)

.sub.iT.sub.Base.sup.Probe=.sub.iT.sub.Base.sup.6Joint.sub.iT.sub.6Joint.sup.Probe(3)

.sub.iT.sub.Base.sup.6Probe=.sub.iT.sub.Base.sup.6Joint.sub.iT.sub.6Joint.sup.ProbeT.sub.6Joint.sup.Tool.sup.1(4)

[0075] FIG. 5 shows examples of basic parts of the algorithm which can be implemented in software. The the left, a state diagram for the learning part is shown, and to the right, a state diagram for the execution part is shown, until (user) stop STP.

[0076] The learning part, to the left, comprises a main loop while the learning is active: #1: Loop start. #2: Read from analog to digital converter data from position joystick J1, rotation joystick J2, and probe sensor PS. #3: Calculate a transformation from the probe sensor PS equation (1) with a Feedback-Assisted algorithm to avoid outer mechanic limits for the probe sensor PS that adjusts the three probe sensor PS axes so they will be close to a center position during the learning process. Calculate transformations based thereon according to equations (1) and (2), and save them. #4: Determine probe sensor correction. #5: Calculates the new transformation from equation (2) with adjustment calculated in state #4, and send it to the robot controller CTL and return to state #1.

[0077] The execution part, to the right, comprises two states in a main loop until (user) stop STP. #6: Calculate equation (4) and send it to the robot controller CTL and return to state #6, as long as there is recorded transformation left.

[0078] FIGS. 6a, 6B, and 6c show photos of a prototype of an implementation of the invention on a CRS robot with two joysticks J1, J2, and a probe sensor PS mounted thereon. An A/D board is used for logging the analog signals from the joysticks J1, J2 and probe sensor PS into a separate computer with the regulation software running. However, as mentioned, a modern robot controller and program can directly log the data without the need for a separate computer.

[0079] A real-time learning session may begin with the robot being controlled to start with a tip of the probe sensor mounted on its TCP so as to start on one end of the desired trajectory to be followed. In a continuous real-time manner, the robot is then controlled to follow the trajectory by simultaneously controlling movement of position in space and orientation of the TCP by the user operating e.g. joysticks, and at the same time, the probe sensor provides input to the robot controller regarding distance and orientation deviation between a neutral position of the probe sensor tip and the actual position of the probe sensor tip. This is performed until the end E_T2 of the desired trajectory has been reached. During the controlling, a logging of data from the robot controller is performed with a suitable temporal and spatial precision in response to the user's operation of the user interface during the learning session. Finally, a calculation of a transformation is performed, so as to adapt the robot movement to a process tool with a given length, and this transformation calculation is performed in response to data logged from the probe sensor.

[0080] FIG. 7 illustrates steps of a method embodiment for controlling a robot during trajectory learning of the robot by demonstration from a user to make the robot follow a desired trajectory during a real-time learning session. The method comprising performing all of the following steps in a continuous manner simultaneously during the real-time learning session: [0081] receiving R_UI user input from at least a first control element for controlling movement of position in space of the robot by operation by the user during the real-time learning session, such as continuously receiving said input during the real-time learning session, [0082] receiving R_PS_D distance input indicative of a distance from the TCP of the robot and a surface forming the trajectory to be followed during the real-time learning session, such as continuously receiving said distance input from a probe sensor, [0083] receiving R_PS_O orientation input indicative of an orientation of the TCP relative to the surface forming the trajectory to be followed during the real-time learning session, such as continuously receiving said orientation input from a probe sensor, [0084] continuously controlling C_RB the robot in response to the user input received user input, the distance input and the orientation input, during the real-time learning session, and [0085] logging data LG_D in response to the user's operation of the at least first control element during the learning session, such as continuously logging said data at a predetermined sample rate, so as to allow later control of the robot in response to the data logged during the learning session.

[0086] This method can be programmed as a control algorithm to form part of a robot controller software package, or it can be fully or partly implemented as a dedicated computer device.

[0087] FIG. 8 shows the analog input from the sensor probe, where a potentiometer for each of X, Y which are angle potentiometers, and Z is a linear potentiometer. The signals are applied to analog-to-digital converters (ADCs). The digital number proportional to the magnitude of the voltage (e.g. 0 to 3.3/5 V interval), then used for calculating the actual angles (X, Y) and the length (Z). The probe needs to be calibrated for finding/calculating the relationship between voltage [V] and angle [radians]. The position is calculated in the last part of the diagram in FIG. 8, and used in the trajectory generator.

[0088] FIG. 9 shows a preferred control algorithm or system with the purpose of controlling position of the tool center point of the robot in response to position feedback from the probe sensor. This is preferably done, so that the robot will move its tool center point in response to the position feedback from the probe sensor, such that the probe sensor will be near its center position at least most of the time during a real-time learning session, irrespective of the user controlled movements of the robot. It is to be understood that a damping is preferred, so as to ensures smooth robot movements without any disturbing feedback effects on the user his/her control of the robot during the real-time learning session. E.g. such automatic robot motion towards the goal of the probe sensor being in its center position may be implemented such that the robot will only seek to obtain the center position of the probe sensor, when the user pauses, e.g. removes hands from the control elements, in the real-time learning session. At least the robot may only seek to obtain the center position of the probe sensor when the user controls the robot to perform movements slower that a certain preset threshold speed. This behavior during a real-time learning session will ensure that the user can more easily maneuver the tip of the probe sensor along a trajectory without unintendendly reachin the limits of the probe sensor, where it is no longer possible to keep the tip of the probe sensor at the preferred trajectory to be taught to the robot.

[0089] Each potentiometer (X, Y, Z) in the probe sensor has a center position defined, meaning that the two angle potentiometers (X, Y) will be in center position, and the linear potentiometer (Z) will be partly pressed in, e.g. half way pressed in between its two outer positions. During the real-time learning process, the trajectory generator will try to keep this center position for the probe sensor, and therefore the trajectory generator will add this to the rest of the control system.

[0090] The Trajectory Generator measures the current probe sensor position and calculates a new relative robot position as output. The system will force the robot to move in a direction, so the next sensor position will be closer to the neutral or center position of the probe sensor. This behaving will avoid any exceed of the probe sensor position limits. Furthermore, this behaving will be the primary control loop and help the user controlling the robot during the real-time continuous trajectory learning session, where the system logs or records the positions and/or control signals at a predetermined sample rate throughout the learning session.

[0091] FIG. 10 shows a photo of a probe sensor embodiment with a spherically shaped tip (a small ball) arranged at an end of a rod, and a protective bellow. The rod is longitudinally displaceable so as to allow variation of distance between tip and base of the probe sensor. The base opposite the tip is arranged for mounting on the tool center point of the robot. The probe sensor has inside the protective bellow two angular potentiometers to sense respective X and Y angles, and one linear potentiometer to sense a length Z of the longitudinally displaceable rod. It may have analog outputs, or a circuit with ADCs to convert to a digital X, Y, Z output in a predetermined digital format at a sample frequency of such as 10-100 Hz, or another suitable sample frequency depending on the equipment used. Such circuit may be arranged inside the bellow in order to allow a digital output signal from the probe sensor.

[0092] To sum up: the invention provides a robot learning system for trajectory learning of a robot RB having a robot arm between a base and a tool center point TCP. A user interface allows the user to control the robot arm in order to follow a desired trajectory during a real-time. A probe sensor PS is mounted on the TCP during the learning session. The probe sensor PS measures a distance parameter Z indicative of distance from the TCP and a surface forming the trajectory to be followed, and an orientation parameter X, Y indicative of orientation of the TCP and the surface forming the trajectory to be followed. These distance and orientation data are provided as a feedback to the controller of the robot CTL during the real-time learning session, thereby allowing the robot controller software to assist the user in following a desired trajectory in a continuous manner. Especially, the probe sensor PS may have a displaceable tip TP to follow a surface and having a neutral or center position, and where the robot controller software controls the robot movements to seek the neutral or center position irrespective of the user's control inputs. Data DT is logged during the learning session, so as to allow later control of the robot RB in response to the data DT logged during the learning session.

[0093] Although the present invention has been described in connection with the specified embodiments, it should not be construed as being in any way limited to the presented examples. The scope of the present invention is to be interpreted in the light of the accompanying claim set. In the context of the claims, the terms including or includes do not exclude other possible elements or steps. Also, the mentioning of references such as a or an etc. should not be construed as excluding a plurality. The use of reference signs in the claims with respect to elements indicated in the figures shall also not be construed as limiting the scope of the invention. Furthermore, individual features mentioned in different claims, may possibly be advantageously combined, and the mentioning of these features in different claims does not exclude that a combination of features is not possible and advantageous.