METHOD AND APPARATUS FOR GENERATING ROBOT PATH DATA TO AUTOMATICALLY COAT AT LEAST PART OF A SURFACE OF A SPATIAL SUBSTRATE WITH AT LEAST ONE COATING MATERIAL

Abstract

Disclosed herein are a method for generating robot path data for robot path(s) to be followed by a robot including a coating tool during coating of at least part of the surface of a spatial substrate with at least one coating material spatial substrate, as well as respective apparatuses, or computer elements. Further disclosed is a robotic system for coating at least one surface of a spatial substrate with at least one coating material. The methods, respective apparatuses, or computer elements allow automated application of coating materials to substrates having a high variation in geometry and provide consistency of application in contrast to manual application of coating materials, for example during repair processes of automotives or automotive parts, which is highly dependent on the painter performing the application.

Claims

1. A computer-implemented method for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating at least part of the surface of at least one spatial substrate with at least one coating material, said method comprising: (a) providing via a communication interface to at least one computer processor spatial substrate data for each spatial substrate including substrate classification data and data being indicative of the geometry and the color of each spatial substrate, and coating material data including data being indicative of the type of the at least one coating material and optionally of the order of the coating materials to be applied to the spatial substrate and/or data being indicative of the coating tool; (b) optionally determining with the at least one computer processor whether each spatial substrate comprises at least one masking material based on the provided spatial substrate data; (c) retrieving via the communication interface using the at least one computer processor, coating tool parameter data based on the provided coating material data, said coating tool parameter data including coating tool tolerance data and at least one application parameter associated with the coating tool, coating procedure data including a rule set for coating outer edges and edges adjacent to open space(s) and a rule set for coating main surfaces, and substrate type data based on the provided spatial substrate data, said substrate type data including a rule set for the type of spatial substrate matching the substrate classification data; (d) generating with the at least one computer processor tool path data for tool path(s) to be followed by the coating tool along the surface of each spatial substrate based on the data retrieved in step (c) and optionally the result of the determination performed in step (b); (e) generating with the at least one computer processor the robot path data based on tool path data generated in step (d); and (f) providing the generated robot path data via the communication interface.

2. The method of claim 1, wherein the data being indicative of the geometry and color of each spatial substrate includes data representing each spatial substrate in three dimensional space.

3. The method of claim 1, wherein providing the spatial substrate data includes detecting, with the at least one computer processor, a user input being indicative of a substrate classification associated with each spatial substrate and a user input being indicative of the location of each spatial substrate within the workspace of the robot, determining, with the at least one computer processor, based on the detected user input, substrate classification data for each spatial substrate and the location of each spatial substrate within the workspace of the robot, providing via a communication interface to the at least one computer processor data of the workspace of the robot, determining, with the at least one computer processor, collision geometries present within said workspace based on the provided data of the workspace of the robot, determining, with the at least one computer processor, scan path data for scan path(s) to be followed by a scanning device along the surface of each spatial substrate based on the determined location of each spatial substrate within the workspace of the robot and the determined collision geometries, and providing, via the communication interface, the determined scan path data to the scanning device, and generating, with the at least one computer processor, the spatial substrate data for each spatial substrate by retrieving, via the communication interface, data being indicative of the geometry and color of the spatial substrate acquired by the scanning device based on the provided scan path data and combining the retrieved data at least with the determined respective substrate classification data.

4. The method of claim 1, wherein data being indicative of the identity of the at least one coating material includes the name of each coating material type, the ID of each coating material type, or a combination thereof.

5. The method of claim 1, wherein the coating tool tolerance data includes target distance data, overlap percentage data, pattern size data, rotational tolerance(s) about the z-axis of the coating tool, rotational tolerance(s) about the x-axis of the coating tool, rotational tolerance(s) about the y-axis of the coating tool, or a combination thereof.

6. The method of claim 1, wherein the rule set for coating outer edges and edges adjacent to open space(s) comprises at least one algorithm for determining outer edges and at least one algorithm for determining edges adjacent to open space(s) present within the surface of the spatial substrate.

7. The method of claim 1, wherein the rule set for coating main surfaces includes rules for determining the start of the coating procedure, rules for coating direction, rules for rotation of coating tool within a tool path, rules for separation of surfaces, or a combination thereof.

8. The method of claim 1, wherein the at least one rule set for the type of spatial substrate matching the substrate classification data contained in the provided spatial substrate data includes at least one rule to coat edges adjacent to open space(s) for the respective type of spatial substrate, data on the required quality of the tool path(s), optionally at least one rule to coat open space(s) within the spatial substrate and optionally at least one rotational tolerance of the coating tool.

9. The method of claim 1, wherein generating tool path data for tool path(s) to be followed by the coating tool along the surface of each spatial substrate includes: generating, with the at least one computer processor, a 3D model of each spatial substrate based on the provided spatial substrate data and optionally applying a rule set to smooth the surface of each generated 3D model, determining, with the computer processor, the outer edge(s), the edge(s) adjacent to open spaces, the main surfaces, and the open space(s) present within each spatial substrate based on the retrieved coating procedure parameter data, and the generated and optionally smoothed 3D model(s), generating, with the at least one computer processor, tool path data for outer edges and tool path data for edges adjacent to open space(s) based on the retrieved coating procedure data, the retrieved coating tool parameter data, the retrieved substrate type data, the determined outer edge(s), edge(s) adjacent to open spaces and open space(s), and the generated and optionally smoothed 3D model(s), generating, with the at least one computer processor, tool path data for main surfaces of each spatial substrate based on the retrieved coating procedure data, the retrieved coating tool parameter data, the determined main surfaces, and the generated and optionally smoothed 3D model(s), and optionally repeating said steps for at least one further coating material based on the retrieved coating procedure data and the retrieved coating tool parameter data associated with the at least one further coating material.

10. The method of claim 1, wherein generating robot path data includes determining collision geometries within the workspace of the robot based on the spatial substrate data and determining robot path data based on the determined collision geometries and the generated tool path data.

11. The method of claim 1, wherein generating robot path data further includes sorting, with the at least one computer processor and prior to generating robot path data, the generated tool path data such that the robot path(s) generated from the tool path data for outer edges and edges adjacent to open space(s) are performed prior to or after the robot path(s) generated from the tool path data for main surfaces and/or optimizing, with the computer processor, the generated or sorted robot path data.

12. A computing apparatus for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating a spatial substrate with at least one coating material comprising: at least one computer processor; and a memory storing instructions that, when executed by the processor, configure the apparatus to perform the steps of claim 1.

13. A robotic system for coating at least one surface of a spatial substrate with at least one coating material, said system comprising: a computing apparatus according to claim 12 for generating robot path data for robot path(s) to be followed by a robot of the robot system during coating the at least one surface of the spatial substrate with at least one coating material, and a robot apparatus configured to receive the generated robot path data and use the received robot path data to apply at least one coating material from a coating tool to the at least part of the surface of the spatial substrate.

14. A method of using the computer-implemented method of claim 1, the method comprising using the computer-implemented method for coating at least part of the surface of a spatial substrate with a coating material using a robotic system comprising a robot containing a coating tool.

15. A non-transitory computer-readable storage medium, including instructions that when executed by a computer, cause the computer to perform the steps according to the method of claim 1.

16. The method of claim 1, wherein the data being indicative of the geometry and color of each spatial substrate includes data representing each spatial substrate in a three-dimensional point cloud of each spatial substrate, as well as color data of each spatial substrate.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0157] These and other features of the present invention are more fully set forth in the following description of exemplary embodiments of the invention. To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced. The description is presented with reference to the accompanying drawings in which:

[0158] FIG. 1 illustrates a flow diagram of a first embodiment of the inventive method for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating at least part of the surface of the spatial substrate with at least one coating material,

[0159] FIG. 2 illustrates a flow diagram of an embodiment for generating spatial substrate data described in relation to block 102 of FIG. 1 in accordance with implementations of the invention,

[0160] FIGS. 3A,B illustrate a flow diagram of an embodiment for generating tool path data described in relation to block 112 of FIG. 1 in accordance with implementations of the invention,

[0161] FIG. 4 illustrates a flow diagram of an embodiment of block 114 of FIG. 1 in accordance with implementations of the invention,

[0162] FIG. 5 illustrates a flow diagram a second embodiment of the inventive method for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating at least part of the surface of the spatial substrate with at least one coating material,

[0163] FIG. 6 illustrates a block diagram of a computing apparatus for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating at least part of the surface of the spatial substrate with at least one coating material in accordance with implementations of the invention,

[0164] FIG. 7 illustrates a block diagram of a robotic system for coating at least one surface of a spatial substrate with at least one coating material in accordance with implementations of the invention,

[0165] FIG. 8A illustrates the z-axis of a spray applicator of a coating tool in accordance with implementations of the invention,

[0166] FIG. 8B illustrates the x- and the y-axis of the spray applicator of the coating tool of FIG. 8A in accordance with implementations of the invention,

[0167] FIG. 9 Illustrates a schematic drawing of a spray booth containing the inventive robotic system in accordance with implementations of the invention,

[0168] FIG. 10 illustrates a top view of the spray booth of FIG. 9 in accordance with implementations of the invention,

[0169] FIG. 11 illustrates a side view of the part of the spray booth of FIG. 9 in accordance with implementations of the invention,

[0170] FIG. 12A illustrates a side view of part of the spray booth of FIG. 9 comprising a tool rack containing a plurality of coating tools and an enclosure housing a sensor system, the coating tools and the sensor system being connectable to the robot of FIG. 11 using a tool changer in accordance with implementations of the invention,

[0171] FIG. 12B illustrates a zoom-in on the tool rack and the enclosure housing the sensor system shown in FIG. 12A.

DETAILED DESCRIPTION

[0172] The detailed description set forth below is intended as a description of various aspects of the subject-matter and is not intended to represent the only configurations in which the subject-matter may be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject-matter. However, it will be apparent to those skilled in the art that the subject-matter may be practiced without these specific details.

[0173] In one case, the illustrated separation of various parts in the figures into distinct units may reflect the use of corresponding distinct physical and tangible parts in an actual implementation. Alternatively, or in addition, any single part illustrated in the figures may be implemented by plural actual physical parts. Alternatively, or in addition, the depiction of any two or more separate parts in the figures may reflect different functions performed by a single actual physical part.

[0174] Other figures describe the concepts in flowchart form. In this form, certain operations are described as constituting distinct blocks performed in a certain order. Such implementations are illustrative and non-limiting. Certain blocks described herein can be grouped together and performed in a single operation, certain blocks can be broken apart into plural component blocks, and certain blocks can be performed in an order that differs from that which is illustrated herein (including a parallel manner of performing the blocks). In one implementation, the blocks shown in the flowcharts that pertain to processing-related functions can be implemented by the hardware logic circuitry described in relation to FIG. 6, which, in turn, can be implemented by one or more Hardware processors and/or other logic components that include a task-specific collection of logic gates.

[0175] As to terminology, the phrase configured to encompasses various physical and tangible mechanisms for performing an identified operation. The mechanisms can be configured to perform an operation using the hardware logic circuitry described in relation to FIG. 6. The term logic likewise encompasses various physical and tangible mechanisms for performing a task. For instance, each processing-related operation illustrated in the flowcharts corresponds to a logic component for performing that operation. A logic component can perform its operation using the hardware logic circuitry as described in relation to FIG. 6. When implemented by computing equipment, a logic component represents an electrical component that is a physical part of the computing system, in whatever manner implemented.

[0176] Any of the storage resources described herein, or any combination of the storage resources, may be regarded as a computer-readable medium. In many cases, a computer-readable medium represents some form of physical and tangible entity. The term computer-readable medium also encompasses propagated signals, e.g., transmitted or received via a physical conduit and/or air or other wireless medium, etc. However, the specific term computer-readable storage medium expressly excludes propagated signals per se, while including all other forms of computer-readable media.

[0177] The following explanation may identify one or more features as optional. This type of statement is not to be interpreted as an exhaustive indication of features that may be considered optional; that is, other features can be considered as optional, although not explicitly identified in the text. Further, any description of a single entity is not intended to preclude the use of plural such entities; similarly, a description of plural entities is not intended to preclude the use of a single entity. Further, while the description may explain certain features as alternative ways of carrying out identified functions or implementing identified mechanisms, the features can also be combined together in any combination. Finally, the terms exemplary or illustrative refer to one implementation among potentially many implementations.

[0178] FIG. 1 illustrates a flow diagram of a first non-limiting embodiment of a method 100 for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating at least part of the surface of at least one spatial substrate with at least one coating material, said method being implemented by a computing device comprising a computer processor. Method 100 may be used for coating at least part of a surface of a spatial substrate with a coating material using a robotic system comprising a robot containing coating tool. The computing device may be the computing system described in relation to FIG. 6 below and may be a stationary device, such as a stationary computer preferably comprising a computer monitor, or a mobile computing device or may be located in a cloud computing environment. The spatial substrate to be coated may be an automotive part, such as an automotive body part, for example an automotive fender. The spatial substrate to be coated may be an automotive hood, an automotive door and automotive bumper cover or any other spatial automotive substrate. A plurality of spatial substrates may be coated which may be of the same substrate type or of a different substrate type. Hence, a plurality of spatial substrates with various geometric shapes may be coated. The automotive fender may comprise a conversion layer but may be free of further coating layers. The automotive fender may already comprise at least one primer, primer-surfacer or basecoat layer. A colored basecoat material and a clearcoat material may be applied by the robot comprising the coating tool. More or less coating materials may be applied by the robot. The robot may be located within a spray booth, for example as described in relation to FIGS. 7 to 11 below.

[0179] In block 102, spatial substrate data of each spatial substrate and coating material data may be provided to the computing device implementing method 100. The provided spatial substate data may at least comprise or include substrate classification data and data being indicative of the geometry and the color of the spatial substrate, in particular the surface of the spatial substrate. In this example, the spatial substrate data includes data representing the spatial substrate in 3D space, in particular 3D point cloud data of the spatial substrate, as well as color data of the spatial substrate, in particular RGB data or static image data acquired by the scanning device (or sensor system) described in relation to FIG. 7 later on. In this example, the spatial substrate data further comprises data being indicative of the identity of the substrate, such as the substrate ID, a substrate name, a bar code, a QR code, as well as a rule set to smooth the surface of a 3D model generated from data being indicative of the geometry and color of the spatial substrate.

[0180] The spatial substrate data may be provided as described in relation with FIG. 2 below. The spatial substrate data may be retrieved from a data storage medium, such as the internal storage of the computing device implementing method 100 or a database connected via a communication interface with the computing device. The stored spatial substate data may be interrelated with a substrate ID to allow retrieval of the spatial substrate data using the substrate ID. The substrate ID may be entered by the user via a graphical user interface allowing the user to enter a substrate ID or select the substrate ID from a list of substrate IDs for which spatial substrate data is available.

[0181] The coating material data comprises data being indicative of the identity of the coating material. The data being indicative of the identity of the coating material may include the name of each coating material type, i.e. basecoat and clearcoat. The data being indicative of the identity of the coating material may include an ID for each coating material type. Providing the coating material data may include displaying a user interface by the computing device allowing the user to select a coating material type from a displayed list of available types. The user interface may allow the user to type the ID or name of each coating material type. The user interface may allow the user to provide the ID/bar code/QR code of the coating material to be applied by the coating tool of the robot and the computing device may retrieve data being indicative of the identity of the coating material from a data storage medium based on the provided ID/bar code/QR code.

[0182] The coating material data may further include data being indicative of the identity of the coating material, such as the ID, a bar code, a QR code, property data of the coating material, such as chemical and/or physical property data, data on the composition of each coating material, or a combination thereof. The coating material data may further include data being indicative of the spatial substrate to be coated with the coating material(s). This may be especially preferred if a plurality of spatial substrates is to be coated with different coating materials to ensure that the correct coating material(s) are applied on each spatial substrate. The property data and data on the composition may be stored on a data storage medium and may be retrieved by the computing device using the provided data being indicative of the identity of the coating material.

[0183] The order of the basecoat and clearcoat to be applied with the coating tool of the robot may not be provided by the user but may be determined by the computing device using a rule set stored on a data storage medium connected to the computing device. The order of the basecoat and clearcoat to be applied with the coating tool of the robot may not be provided by displaying a user interface prompting the user to select the order of the provided coating material types.

[0184] Data being indicative of the coating tool may not be provided by the user but may be retrieved from a data storage medium based on the provided coating material type or coating material ID. Data being indicative of the coating tool may be provided by displaying a user interface prompting the user to select the coating tool, for example by displaying a list of available coating tools.

[0185] In block 104, method 100 may determine whether the surface of the spatial substrate comprises at least one masking material based on the spatial substrate data provided in block 102, this step being generally optional. Block 104 may be performed to reduce consumption of the applied coating materials and to avoid overspray because areas covered with masking material may be excluded when determining tool path data and thus robot path data to avoid application of coating material onto the masking material. Block 104 may be performed by the computing device performing the other blocks of method 100 or by a further computing device, such as a server device being connected to the computing device via a communication interface and having access to a trained machine learning model used to determine the presence of masking materials on the spatial substrate. In this case, the computing device may function as client device and may provide the spatial substate data to the server device. The server device may then use the provided spatial substate data and the trained machine learning model to determine the presence of masking material and may provide the result of the determination to the computing device. Use of a further server device to determine the presence of masking materials may be beneficial because the trained machine learning model can be stored in a database only accessible to the server, thus reducing the overall system complexity. Moreover, the feedback of the user concerning the correct determination of the masking materials may be used by the server device to improve the trained machine learning model using commonly known learning techniques.

[0186] Determining whether the spatial substrate comprises at least one masking material based on the provided spatial substrate data may include generating a three-dimensional (3D) model using the provided spatial substrate data and providing the generated 3D model to a trained machine learning model, in particular a deep learning algorithm. The 3D model may be created using the 3D point cloud data and the color data contained in the provided spatial substrate data, i.e. the 3D model is a 3D color model. Generation of the 3D model from the point cloud data and color data may be performed according to methods well known in the state of the art.

[0187] The machine learning model may have been trained on historical data of masking materials, in particular on historical 3D color models of spatial substrates containing masking material(s), to determine the presence of masking material(s) on the surface of spatial substrates based on provided historical spatial substrate data. The machine learning model, in particular the deep learning algorithm, may be hosted by the computing device implementing method 100, a remote server or a cloud or other server. Advantageously, by locating the algorithm on a remote server or a cloud server, costs of added memory and/or a more complex processor in using the algorithm to determine the presence of masking material(s) on the surface of each spatial substrate can be avoided. Additionally, continuous, or periodic improvement of the algorithm can more easily be done on a centralized server and avoid data costs and risks of pushing out an update of the algorithm to each computing device. A remote server may also serve as a central repository storing training and/or collections of operative data sent from various computing devices to be used to train and develop existing algorithms. For example, a growing repository of data can be used to update and improve algorithms on existing systems and to provide improved algorithms for future use.

[0188] The trained machine learning algorithm used in block 104, more specifically the artificial neural network (ANN) model, may be obtained using commonly known machine learning methods to train the ANN model to determine the presence of masking materials on 3D color models of spatial substrates. An exemplary commercially available software to implement the training process is Keras (available on the Internet at Keras.io), an open source ANN model library that runs on top of either TensorFlow or Theano, which provide the computational engine required. TENSORFLOW (an unregistered trademark of Google, of Mountain View, Calif.) is an open source software library originally developed by Google of Mountain View, Calif. and is available as an internet resource at www.tensorflow.org.

[0189] The model training data sets used for training of the machine learning model may be divided into three portions: the training set, the validation set, and the verification (or testing) set. The training set is used to adjust the internal weighting algorithms and functions of the hidden layers of the neural network so that the neural network iteratively learns how to correctly recognize and classify patterns in the input data. The validation set, however, is primarily used to minimize overfitting. The validation set typically does not adjust the internal weighting algorithms of the neural network as does the training set, but rather verifies that any increase in accuracy over the training data set yields an increase in accuracy over a data set that has not been applied to the neural network previously, or at least the network has not been trained on it yet (i.e. validation data set). If the accuracy over the training data set increases, but the accuracy over then validation data set remains the same or decreases, the process is often referred to be overfitting the neural network and training should cease. Finally, the verification set is used for testing the final solution in order to confirm the actual predictive power of the neural network.

[0190] In one example, approximately 70% of the developed or collected data model sets are used for model training, 15% are used for model validation, and 15% are used for model verification. These approximate divisions can be altered as necessary to reach the desired result. For example, about 300 sets of data may be collected, each set including 3D models of spatial substrates comprising masking material(s) and being free of masking materials. The training data set may include samples throughout a full range of expected spatial substrates and positions and types of masking material(s) on said substrates.

[0191] The result of the determination may be provided to the computing device and the computing device may display the result of the determination in a graphical user interface to allow the user to check whether masking materials present on the spatial substrate have been recognized correctly. The provided result may represent an overlay of the 3D model processed by the machine learning model on top of the unprocessed 3D model (i.e. the 3D model generated from the spatial substrate data). A 3D color model comprising a classification of which surface areas contain masking material(s) may be provided to the display device for display on the screen, for example within a graphical user interface.

[0192] This allows the user to check whether the presence and location of masking material(s) was correctly detected by the trained machine learning model and allows to correct the presence and location if the masking material(s) present on the spatial substrate were not correctly determined.

[0193] In block 106, the computing device implementing method 100 may retrieve coating tool parameter data based on the coating material data provided in block 102. If more than one coating material is to be applied, i.e. the coating material data provided in block 102 may contain data on more than one coating material type, coating tool parameter data associated with each coating material type contained in the provided coating material data may be retrieved in block 106. The coating tool parameter data may include coating tool tolerance data and at least one application parameter associated with the coating tool used by the robot to coat at least part of the surface of the spatial substrate with the respective coating material. The coating tool parameter data may be stored in a database connected to the computing device via a communication interface and data contained in the coating material data provided in block 102, such as the name or ID of the type of coating materials to be applied, may be used by the computing device to retrieve the appropriate coating tool parameter data. Retrieving coating tool parameter data based on the coating material data provided in block 102 may include retrieving data being indicative of the type of the coating materials (i.e. basecoat and clearcoat) contained in the provided coating material data and retrieving the coating tool parameter data associated with the basecoat and the clearcoat.

[0194] The coating tool tolerance data may include target distance data, overlap percentage data, pattern size data, rotational tolerance(s) about the z-axis of the coating tool, rotational tolerance(s) about the x-axis of the coating tool and rotational tolerance(s) about the y-axis of the coating tool. Target distance data and overlap percentage may refer to a defined target distance value/overlap percentage and/or a target distance range/overlap percentage range. For example, the target distance data may comprise a range from 4 to 8 inches and the overlap percentage data may comprise a range of 70 to 90% for basecoat materials. For clearcoat materials, the target distance data may comprise a range of 6 to 10 inches and the overlap percentage data may comprise a range of 40 to 60% The target distance data and overlap percentage data may each comprise a defined value, such as 6 inches for basecoat materials and 8 inches for clearcoat materials, as well as an allowable range to allow to generate appropriate tool path data. Combination of a defined value and a range allows to use the defined value as a starting point for generation of tool path data and to adapt said starting point with the provided range if it is required to generate tool path data having the required quality. The overlap percentage data may remain fixed while other parameters may be adjusted during generation of the tool path data. The overlap percentage data may be adjusted during generation of the tool path data. The pattern size data may be fixed for a certain target distance and may scale with said target distance. For example, if one assumes a 10 inch tall and 1.5 inch wide pattern for a target distance of 6 inch and a 14 inch tall and 2.5 inch wide pattern for a target distance of 12 inch, one can use a linear interpolation between these two points to obtain the pattern size data in between these two points. The pattern size data may comprise rules that correlate the given target distance with a certain pattern size or pattern size range. The rotational tolerance about the z-axis (roll) as well as of the y-axis (yaw) of the coating tool may be within 15 of the normal surface vector and the rotational tolerance about the x-axis (pitch) of the coating tool may be within 5 of the normal surface vector.

[0195] The at least one application parameter associated with the coating tool may include the flow rate of the coating material, the voltage applied to the coating material, the pressure applied to the coating material and the bell speed of the coating tool. This may be preferred if the coating tool corresponds to an ESTA (electrostatic spray application) applicator. The at least one application parameter associated with the coating tool may include the flow rate of the coating material and the pressure applied to the coating material. This may be preferred if the coating tool corresponds to a pneumatic coating material applicator.

[0196] In block 108, the computing device implementing method 100 may retrieve coating procedure data (CPD). The retrieved coating procedure data may include a rule set for coating outer edges and edges adjacent to open space(s) and a rule set for coating main surfaces. The rule set for coating outer edges of the spatial substrate and edges adjacent to open space(s) may comprise at least one algorithm for determining outer edges and at least one algorithm for determining edges adjacent to open space(s) present within the surface spatial substrate. The algorithm for determining edges adjacent to open space(s) may further allow to determine open space(s) present with the spatial substrate which are at least partly surrounded by the determined edges adjacent to open space(s). The rule set for coating outer edges and edges adjacent to open space(s) may include rules for coating edges adjacent to open space(s), rules for coating outer edges and optionally rules of coating open space(s). At least the rules for coating edges adjacent to open space(s) and the rules of coating open space(s) may include coating tool tolerance data being different from the coating tool tolerance data contained in the retrieved coating tool parameter data. This allows to modify the retrieved coating tool tolerance data in case edges adjacent to open space(s) and thus also open space(s) are detected and thus allows to use different target distances and/or overlap percentages and/or pattern sizes and/or rotational tolerance(s) for coating open space(s) and their edges as well as outer edges.

[0197] The rule set for coating main surfaces may include rules for determining the start of the coating procedure, rules for coating direction, rules for rotation of coating tool within a tool path and rules for separation of surfaces. The rule set for coating main surfaces may include more or less rules. Rules to determine the start of the coating procedure may, for example, include determining a corner of the substrate and proceeding in a direction allowing to maintain a vertical orientation of coating material reservoir of the coating tool. Rules for coating direction may, for example, include coating the substrate from top-bottom or bottom-up and/or limiting tool path(s) in curvature relative to the edges of the substrate such that the tool path(s) do not follow the curvature or only follow the curvature to a certain extend. Rules for rotation of the coating tool within a tool path may include, for example, limiting the rotation such that the coating tool does not change orientation during painting a surface (or target area) of the spatial substrate. Rules for separation of surfaces may include determining separate surfaces and coating separated, i.e. adjacent, surfaces. The presence of separate surfaces (or different target areas) may be determined by determining whether the spatial substrate includes surfaces have a certain relative angularity between the resulting two faces and radius of curvature. For example, separation of surfaces is given if the determined relative angularity is from 25 to 90 and the radius of curvature is from 1 to 10 inches. Ensuring that both ranges for relative angularity and radius of curvature are fulfilled ensures that small body lines with a small radius of curvature but little difference in angularity between the two resulting faces are determined to be separate surfaces, thus avoiding unnecessary separation of surfaces of the spatial substrate. Coating separate surfaces may include coating the adjacent surfaces in an adjacent-next pattern to maintain a wet film on both adjacent surfaces. This ensures that the freshly coated surface is wet to accept overspray, and the adjacent surface to be coated has still wet overspray when it is coated.

[0198] The retrieved coating procedure data may further include a rule set for application of at least two different coating materials on the same spatial substrate. This may be preferred if the coating process allows to apply more than one coating material to the same spatial substrate, such as a basecoat material followed by a clearcoat material. The coating procedure data may not include said rule set, for example if the coating process allows to only apply a single coating material to the same spatial substrate. The rule set may contain rules for copying coating tool tolerance data used to apply the previous coating material (in this case the basecoat material) and to modify the copied data based on the coating material data provided in block 102. Moreover, the rule set may include rules to eliminate or perform determination of edges adjacent to open space(s) and thus open space(s). This allows to either coat open space(s) with the further coating material (in this example the clearcoat material) or to avoid coating the open space(s) with the further coating material, for example if a clearcoat layer is not required on the open space(s) coated with a basecoat layer.

[0199] By using rule sets during generation of the robot path data, such robot path data may be reliably generated for substrates having a high level of geometric variation This allows to coat such substrates by the robotic system using the generated robot path data such that coatings having a high optical and mechanical quality are obtained irrespective of the geometry of the respective substrate. Hence, the robot path data can be tailored to the respective substrate using such rule sets without requiring manual interaction to ensure that the geometry of the substrate is sufficiently considered during generation of the robot path data to avoid improper coating material application which may result in reduced optical and/or mechanical quality of the resulting coating.

[0200] In block 110, the computing device implementing method 100 may retrieve substrate type data (STD) based on the spatial substrate data provided in block 102. For this purpose, the at least one processor contained in the computing device may determine the substrate classification data contained in the spatial substrate data provided in block 102 and may retrieve the associated substrate type data, for example from a database or the internal memory of the computing device, the retrieved substrate type data may contain at least one rule set including at least one rule to coat edges adjacent to open space(s) for the respective type of spatial substrate, at least one rule to coat open space(s) present within the spatial substrate and at least one rotational tolerance of the coating tool. The at least one rule set may include more or less rules or may not include the rotational tolerance(s) of the coating tool.

[0201] In block 112, the computing device implementing method 100 may generate tool path data based on the retrieved coating tool parameter data, the retrieved coating procedure data, the retrieved substrate type data and the result of the determination of block 104, in case said optional block has been performed. The tool path data may be generated as described in FIGS. 3A and 3B below.

[0202] In block 114, the computing device implementing method 100 may generate robot path data based on the tool path data generated in block 112. Generating the robot path(s) may include determining collision geometries within the workspace of the robot based on the spatial substrate data and determining robot path data based on the determined collision geometries and the generated tool path data. Determination of the collision geometries ensures that the robot does not hit and destroy the spatial substrate(s) during the coating procedure. An initial pose of the robot may be determined and based on that initial pose, all subsequent robot movements may be generated allowing the robot, in particular the movable robot member described in relation to FIG. 7, to follow the generated tool path data as well as to return to the initial pose. The generated tool path data may be sorted prior to generating the robot path data. The generated robot path data may be optimized as described in relation to FIG. 4 below.

[0203] In block 116, the robot path data generated in block 114 may be provided via a communication interface. This may include providing the generated robot path data to a robot controller connected via the communication interface with the computing device as well as with a robot as described in relation to FIG. 7 later on. The generated robot path data may further be interrelated with the ID of the spatial substrate and may be provided to a data storage medium, such as a database or internal memory.

[0204] After the end of block 116, method 100 may end or may return to block 102, for example if it detects that spatial substrate data and coating material data are provided.

[0205] FIG. 2 illustrates a flow diagram of a method 200 for generating spatial substrate data described in relation to block 102 of FIG. 1. The generated spatial substrate data may contain a substrate ID as well as a rule set to smooth the surface of a 3D model generated from data being indicative of the geometry and color of each spatial substrate. Method 200 may be performed by the computing device described in relation with FIG. 1, said computing device comprising at least one processor implementing a routine to perform the steps described in relation with the following blocks 202 to 216.

[0206] In block 202, the routine implementing method 200 may detect a user input being indicative of a substrate classification associated with each spatial substrate and a user input being indicative of the location of each spatial substrate within the workspace of the robot. The user input may be detected by displaying a user interface on a display of the computing system implementing method 100, the user interface allowing the user to select the substrate classification for each spatial substrate present within the workspace of the robot, for example by displaying a list of possible substrate classifications, such as trunk, hood, fender, etc., or by displaying a text field and prompting the user to enter the respective substrate classification(s). The user interface may also allow the user to select the location of each spatial substrate within the workspace of the robot, i.e. the spray booth, for example by displaying a map or image of the spray booth and prompting the user to select the location of each spatial substrate in the displayed map of the spray booth. The spray booth may be divided into different zones, such as 5 zones, and the user may be prompted to select all zones comprising the spatial substrate. The zones may be marked in the spray booth to facilitate selection of the zone(s) by the user. The user input may be detected via an interaction element, such as an input device or input/output device, in particular a mouse, a keyboard, a trackball, a touch screen or a combination thereof, which may be attached via a communication interface to the computing device implementing method 200.

[0207] In block 204, the substrate classification data and the location of each spatial substrate within the workspace of the robot may be determined based on the detected user input. If the user input is detected with a touchscreen device, the processor located within the touchscreen device may detect the user input and may determine the substrate classification data and location of each spatial substrate based on the detected user input. The determined data may then be provided to the computing device implementing method 200. If the user input is detected via a mouse, keyboard or trackball attached to the computing device implementing method 200, the substrate classification data and location of the substrate may be determined with the processor(s) housed within said computing device. Blocks 202 and 204 may either be performed prior to blocks 206 to 212, after block 212 or 214 or at any time prior to block 216.

[0208] In block 206, data of the workspace of the robot may be provided via a communication interface to the routine implementing method 200. Data of the workspace of the robot may be provided by scanning the workspace of the robot with a scanning device (or sensor system) attached to a movable robot member of the robot as described in relation to FIG. 7 below. The data of the workspace of the robot may include 3D point cloud data as well as color data, such as RGB or static image data. The computing device implementing method 200 may be used to control the robot comprising the scanning device via a robot controller as described in relation to FIG. 7 below. For example, the computing system may allow the user to issue command(s) regarding the scan of the workspace of the robot via the computing device to the robot controller. Scanning of the workspace of the robot may also be automatically initiated, for example upon starting the inventive method, by issuing a respective command to the robot controller by the computing device. The robot controller may then control the robot comprising the scanning device to scan the workspace and the data acquired by the scanning device may be provided via the robot controller to the computing device.

[0209] In block 208, the routine implementing method 200 may determine collision geometries present within said workspace based on the data of the workspace of the robot provided in block 206. Collision geometries may be generated by filtering the provided data of the workspace of the robot to reduce the number of data points, determining whether said data contains geometries having a certain size, creating 3D object(s) present within the workspace from extreme points and filling the generated object(s) with volume to obtain the collision geometries.

[0210] In block 210, the routine implementing method 200 may determine scan path data for scan path(s) to be followed by a scanning device along the surface of each spatial substrate based on the determined location of each spatial substrate within the workspace of the robot and the determined collision geometries, and may provide, via the communication interface, the determined scan path data to the scanning device. If the spatial substrate is present in more than one zone of the spray booth or a plurality of spatial substrates is present within different zones, the zones may be gone through in predefined order during generation of the scan path data to ensure that the complete surface of each spatial substrate is scanned by the scanning device. The determined scan path data may be provided via a communication interface to a scanning device, for example sensor device 706 described in relation to FIG. 7, which may then scan the surface of each spatial substrate based on the provided scan path data. The scanning device may be the same scanning device, or a different scanning device as used to acquire data of the workspace of the robot provided in block 206. The scanning device may be attached to the movable robot member of the robot and the determined scan path data may be provided from the computing device implementing method 200 to the robot controller controlling the robot comprising the scanning device. Performing said scan provides detailed information on the spatial substrate and allows to generate tool path data with results in consistent high optical quality in terms of appearance of the resulting coating.

[0211] In block 214, the routine implementing method 200 may retrieve the data being indicative of the geometry and color of each spatial substrate which has been acquired by the scanning device based on the scan path data determined and provided to said scanning device in block 212. For this purpose, the acquired scan path data may be assigned to a location ID and the routine may associate the location ID with the determined location of each spatial substrate to assign the acquired scan path data to the respective spatial substrate. The retrieved data may include 3D point cloud data and RGB color data or statical image data. The acquired scan data may be stored by the scanning device or by the controller controlling the scanning device, such as the robot controller or a further computing device and said data may be retrieved from said storage. The acquired scan data may be provided in real time to the computing device implementing method 200, which may store the received scan data.

[0212] In block 216, the routine implementing method 200 may generate spatial substrate data for each spatial substrate by combining the data retrieved in block 214 with the respective determined substrate classification data, the respective data being indicative of the identity of the spatial substrate and the respective rule set to smooth the surface of the generated 3D model. The rule set may be retrieved using the substrate classification data determined in block 204, for example by using a database containing rule sets, each rule set being interrelated with appropriate substrate classification data. The spatial substrate data for each spatial substrate may be generated by combining the data retrieved in block 214 with the respective determined substrate classification data. The spatial substrate data generated in block 214 for each spatial substrate may be stored on a data storage medium, such as the internal memory, prior to using said data in further blocks described in relation to FIG. 1. After the end of block 216, method 200 may proceed to block 104 of FIG. 1 described previously.

[0213] FIGS. 3A and 3B illustrate a flow diagram of a method 300 for generating tool path data described in relation to block 114 of FIG. 1. Method 300 may be performed by the computing device described in relation to FIG. 1, said computing device comprising at least one processor implementing a routine to perform the steps described in relation with the following blocks 302 to 326.

[0214] In block 302, the routine implementing method 300 may generate a three-dimensional (3D) model for each spatial substrate based on the spatial substrate data provided in block 102 of FIG. 1. The provided spatial substrate data may have been generated as described in relation to FIG. 2 above. The spatial substrate data may containfor each spatial substrate3D point cloud data, static image data of the spatial substrate and a rule set to smooth the surface of a generated 3D model. Generation of a 3D color model from the 3D point cloud data and the static image data may be performed using commonly known methods and software, such as the open-source library Open3D.

[0215] In block 304, the routine implementing method 300 may retrieve the respective rule set and may apply the retrieved rule set to smooth the surface of the 3D model generated in block 302 for each spatial substrate, this step being generally optional. The rule set may be contained in the spatial substrate data provided in block 102 of FIG. 1. The routine implementing method 300 may retrieve the rule set from a data storage medium, comprising rule sets interrelated with substrate classification data, based on the substrate classification data contained in the provided spatial substrate data. Smoothing of each generated 3D model may involve determining the surfaces (or areas) of the 3D model which match surface data contained in rule(s) of the rule set and applying an optimization algorithm, such as a numerical smoothing algorithm known in the state of the art, in combination with threshold values contained in said rule(s) on the determined surfaces. The threshold values may be used to control the degree of smoothing performed by the optimization algorithm. For example, if the spatial substrate type is a hatch, corners are smoothed more and an interior edge and cavities for taillights are determined whereas on a front quarter panel the corners are kept sharper and any holes that might be treated as internal edges are ignored. Use of the rule set to smooth the 3D model(s) allows to control the degree of smoothing and thus the resulting quality of the coating in terms of appearance, because the higher the degree of smoothing, the lower the resulting quality. A higher degree of smoothing may be used for surfaces, such as front light pockets present within a fender, which do not require a high quality while a lower degree of smoothing may be used for surfaces requiring a higher degree of quality, such as the part of the fender visible after mounting the coated fender to the car body. Smoothing the 3D model(s) allows to generate the tool path data more efficiently, because the meshes of the 3D model(s) obtained from the point cloud data are converted into surfaces, which can be processed more efficiently during generation of the tool path data, for example by the computer processor(s) generating the tool path data as described later on.

[0216] In block 306, the routine implementing method 300 may determine the outer edge(s), the edge(s) adjacent to open space(s), the main surfaces, and the open space(s) present within each spatial substrate based on the coating procedure data retrieved in block 108 of FIG. 1 and each 3D model generated in block 302 or each 3D model smoothed in block 304. For this purpose, the routine may retrieve the appropriate rule set and applies the rule set on each generated/smoothed 3D model. The routine may retrieve the algorithms from the rule set and may apply the algorithms to each smoothed 3D model to determine the outer edges, edges(s) adjacent to open space(s), main surfaces, and open space(s) as described previously.

[0217] In block 308, the routine implementing method 300 may generate tool path data for outer edges and tool path data for edges adjacent to open space(s) based on the coating tool parameter data and the coating procedure data associated with the first coating material (in this example the basecoat material) retrieved in blocks 106 and 108 of FIG. 1, the substrate type data retrieved in block 110 of FIG. 1 and each 3D model generated in block 302 or each smoothed 3D model generated in block 304 The tool path data for outer edges and tool path data for edges adjacent to open space(s) may be generated by [0218] determining the target distance(s), overlap percentage(s), pattern size and rotational tolerance(s) of the coating tool based on the outer edges, the edges adjacent to open space(s) and the open space(s) determined in block 306, the retrieved coating tool parameter data, the retrieved coating procedure data, the retrieved spatial substrate data, and each generated and smoothed 3D model, and [0219] generate tool path data from the determined the target distance(s), overlap percentage(s), pattern size and rotational tolerance(s) of the coating tool using each generated and smoothed 3D model.

[0220] For example if the spatial substrate is a fender or a hood, open space(s) present within the fender or hood, such as marker light pockets within the fender or hood scoop pockets within the hood, may not be treated according to rule(s) for coating open space(s) contained in the substrate type data. The front light pocket as well as the top inner rail and tabs and the lower flange on the dog-leg may be determined as open space(s) which are to be coated according to rules contained in the coating procedure data. If the spatial substrate is a door, open space(s) present within the door-except for windows-such as door handle mounting holes, may not be treated according to rule(s) for coating open space(s) contained in the substrate type data. Door jambs and window frames may be determined as open space(s) which are to be coated according to rules contained in the coating procedure data.

[0221] In block 310, the routine implementing method 300 may generate tool path data for main surfaces of each spatial substrate based on the coating procedure data associated with the first coating material (in this example the basecoat material), in particular the rule set for coating main surfaces, retrieved in block 108 of FIG. 1 and each 3D model generated in block 302 or each smoothed 3D model generated in block 304. Tool path data for main surfaces may be generated by [0222] determining whether main surfaces of each spatial substrate comprise at least two separate surfaces based on the retrieved coating procedure data and each generated and smoothed 3D model, [0223] determining whether main surfaces of each spatial substrate comprise a convex shape and creating an outer hull around said spatial substrate(s) using a convex hull method, [0224] determining the target distance(s), overlap percentage(s), pattern size and rotational tolerance(s) of the coating tool based on the determined separate surfaces, the determined convex shape, the retrieved coating tool parameter data, the retrieved coating procedure data, and each generated and smoothed 3D model, and [0225] generating tool path data from the determined the target distance(s), overlap percentage(s), pattern size, rotational tolerance(s) of the coating tool using each generated and smoothed 3D model.

[0226] The rule set for coating main surfaces may include rules for determining the start of the coating procedure, rules for coating direction, rules for rotation of coating tool within a tool path and rules for separation of surfaces. The rule set for coating main surfaces may comprise more or less rules. Rules to determine the start of the coating procedure may, for example, include determining a corner of the substrate and proceeding in a direction allowing to maintain a vertical orientation of coating material reservoir attached to the spray applicator. Rules for coating direction may, for example, include coating the spatial substrate from top-bottom or bottom-up, such as doors, or from leading edge to back edge or from side to side, such as hoods, and/or limiting tool path(s) in curvature relative to the edges of the substrate such that the tool path(s) do not follow the curvature or only follow the curvature to a certain extend. Rules for rotation of the coating tool within a tool path may include, for example, limiting the rotation such that the coating tool does not change orientation during painting a surface (or target area) of the spatial substrate. Rules for separation of surfaces may include determining separate surfaces and coating separated, i.e. adjacent, surfaces. In this example, the presence of separate surfaces (or different target areas) is determined by determining whether the spatial substrate includes surfaces have a certain relative angularity between the resulting two faces and radius of curvature. For example, separation of surfaces is given if the determined relative angularity is from 25 to 90 and the radius of curvature is from 1 to 10 inches. Ensuring that both ranges for relative angularity and radius of curvature are fulfilled ensures that small body lines with a small radius of curvature but little difference in angularity between the two resulting faces are determined to be separate surfaces, thus avoiding unnecessary separation of surfaces of the spatial substrate. Coating separate surfaces may include coating the adjacent surfaces in an adjacent-next pattern to maintain a wet film on both adjacent surfaces. This ensures that the freshly coated surface is wet to accept overspray, and the adjacent surface to be coated has still wet overspray when it is coated.

[0227] Determination of whether the spatial substrate comprises a convex hull may be performed based on each generated or smoothed 3D model by determining the surface curvature of each spatial substrate from side to side. Such convex surfaces may, for example, the present on bumper covers or on hoods. The outer shell resulting from applying a convex hull algorithm known in the state of the art to the convex surface of said spatial substrate(s) may be used to generated tool path data for said convex surface.

[0228] The steps of determining whether main surfaces of each spatial substrate comprise at least two separate surfaces and/or convex surfaces may be performed in any order.

[0229] In block 312, the routine implementing method 300 may calculate the coating material application for each spatial substrate using the tool path data generated in blocks 308 and 310. Calculation of the respective coting material application using the generated tool path data may include simulation of the coating material application using the generated tool path data. This simulation allows to calculate a coverage map of each generated or smoothed 3D model. The calculated coverage map(s) may be provided via a communication interface, for example to a display device, which displays the received coverage map(s) within a graphical user interface (GUI). The coverage map(s) may comprise different colors to indicate whether the predefined parameter(s) are fulfilled or not. The coverage map(s) may also contain the calculated coating material application as well as the retrieved predefined parameter(s) to allow a visual comparison.

[0230] In block 314, the routine implementing method 300 may determine whether each coating of each spatial substrate resulting from the calculated coating material application on each spatial substrate fulfils at least one predefined parameter. For this purpose, the routine may compare the calculated coating material application for each spatial substrate with each retrieved predefined parameter for each spatial substrate and determines whether the calculated coating material application is inside or outside of the retrieved predefined parameter. The predefined parameters may be retrieved by the routine from a database using the provided material coating data, for example the ID of the coating material or the coating material type. The database may contain said parameter(s) interrelated with the coating material ID or coating material type. The dry and/or wet film thickness or range thereof and/or the surface area to be coated or a range thereof may be retrieved as predefined parameter(s). The coverage map(s) calculated in block 312 may be used to determine whether a certain target area of the 3D model of each spatial substrate will be coated with the appropriate percentage of the coating material such that the wet and/or dry film thickness of the resulting coating layer fulfills the retrieved predefined range of wet and/or dry film thickness. The coverage map(s) may also be used to control the thickness of transparent coating materials or the thickness of coatings on parts of the spatial substrate(s) that require certain specifications with respect to transmissibility, like ADAS (advanced driver assistant system) and radar sensors.

[0231] If at least one predefined parameter is not fulfilled, the routine may return to block 308 and may repeat blocks 308 and 310 using the result of the determination of block 314. After new tool path data has been generated upon repeating blocks 308 and 310, the newly generated tool path data may be checked using the steps described in blocks 312 and 314 above. This loop may be repeated until it is determined in block 314 that the generated tool path data fulfills at least one, in particular all, predefined parameters. Performing blocks 312 to 314 ensures that the coating resulting from applying a coating material using the robot path data generated from said tool path data fulfills predefined quality parameters, such as wet and/or dry film thickness and surface area to be coated. If the routine implementing method 300 determines in block 314 that at least one, in particular all, retrieved predefined parameters are fulfilled, it may proceed to block 316.

[0232] In block 316, the routine implementing method 300 may determine whether a further coating material is to be applied apart from the first coating material. This determination may be made based on the data contained in the coating material data provided in block 102 of FIG. 1. For example, if said data contains data for at least two different coating materials, the routine may use said information to determine the number and type of coating materials to be applied. In case the routine determines that a further coating material is to be applied, it may proceed to block 318. Otherwise, it may proceed to block 116 of FIG. 1 described previously.

[0233] In block 318, the routine implementing method 300 may generate tool path data as described in relation to blocks 308 and 310 based on the coating procedure data and coating tool parameter data associated with the further coating material.

[0234] In block 320, the routine implementing method 300 may calculate the coating material application using the tool path data generated in block 318 as described in relation with block 312.

[0235] In block 322, the routine implementing method 300 may determine whether the coating material application calculated in block 320 fulfils at least one, in particular all, predefined parameters as described in relation to block 316. If at least one predefined parameter is not fulfilled, the routine implementing method 300 may returns to block 318 and may optimize the generated tool path data as described in relation to block 314. If at least one predefined parameter is fulfilled, the routine may proceed to block 324.

[0236] In block 324, the routine implementing method 300 may determine whether at least one further coating material is to be applied as described in relation to block 316. If the routine determines that at least one further coating material is to be applied, it may proceed to block 318 described above, otherwise it may proceed to block 116 of FIG. 1.

[0237] FIG. 4 illustrates a flow diagram of a method 400 for generating robot path data as described in block 114 of FIG. 1. Method 400 may be performed by the computing device described in relation with FIG. 1, said computing device comprising at least one processor implementing a routine to perform the steps described in relation with the following blocks 402 to 410.

[0238] In block 402, the routine implementing method 400 may determine whether to sort the generated tool path data. This determination may be made based on the data contained in the retrieved coating procedure data or based on the programming of the routine. For example, the retrieved coating procedure data may contain rules that determine that tool path data generated for outer edges and edges adjacent to open space(s) may be performed prior to tool path data generated for open space(s) and main surface(s). Performing tool path(s) for outer edges and edges adjacent to open space(s) prior to tool path(s) for open space(s) and main surfaces may be beneficial because coating of the edges generates overspray on the main surfaces which can be covered by coating the main surfaces afterwards such that a negative influence on the final overall appearance is avoided. Moreover, this avoids application of too much coating material on the edges, which is unfavorable because too much coating material present on the edges is prone to sagging, runs and heavy edges, creating a negative influence on the final overall appearance of the coated substrate. If the routine determines in block 402 that the tool path data generated in block 112 of FIG. 1 is to be sorted it may proceed to block 404, otherwise it may proceed to block 406 as described later on.

[0239] In block 404, the routine implementing method 400 may sort the tool path data generated in block 112 of FIG. 1 according to its programming or according to retrieved rule(s). The retrieved rules may be contained in the coating procedure data or may be retrieved by the routine from a data storage medium, such as a database, based on the provided coating material data. The generated tool path data may be sorted according to rule(s) contained in the retrieved coating procedure data and according to rules retrieved based on the provided coating material data. For this purpose, the routine may retrieve the appropriate rules from the retrieved coating procedure data and may apply the rules on the generated tool path data to sort the tool path data accordingly. The generated tool path data may comprise meta data indicating whether the tool path data has been generated for outer edges, edges adjacent to open space(s), open space(s) or main surfaces to facilitate sorting to the generated tool path data using said meta data. If more than one spatial substrate is to be coated, the routine may sort the generated tool path data such that all tool paths associated with a specific coating material type, such as basecoat material, are performed prior to performing all tool paths associated with a further coating material, such as clearcoat material. However, it is also possible to sort the tool path data such that one spatial substrate is completely coated prior to coating a further spatial substrate. The tool paths associated with a specific coating material, such as a primer coating material, may be performed prior to tool paths associated with a basecoat and clearcoat material for each specific spatial substrate, i.e. the tool paths for basecoat and clearcoat material or each spatial substrate may be grouped and performed in sequence such that basecoat and clearcoat material is applied per spatial substrate after applying the primer coating material to all spatial substrates.

[0240] In block 406, the routine implementing method 400 may generate robot path data using the tool path data generated in block 112 as described in relation to block 114 of FIG. 1.

[0241] In block 408, the routine implementing method 400 may determine whether to optimize the generated robot path data. This determination may be made according to the programming of the routine. Optimization of the generated or sorted robot path data may be beneficial because it allows to smoothen the motion of the robot to make it more efficient and consistent. If the routine determines in block 408 that the generated robot path data is to be optimized, it may proceed to block 410. Otherwise it may proceed to block 116 of FIG. 1 as described above.

[0242] In block 410, the routine implementing method 400 may optimize the generated robot path data. Optimization of generated robot path data may be performed, for example, using available open-source libraries, such as Descartes (ROS-Industrial project for performing path-planning on under-defined Cartesian trajectories), and software frameworks, such as trajopt (software framework for generating robot trajectories by local optimization). After the end of block 410, the routine may proceed to block 116 of FIG. 1.

[0243] FIG. 5 illustrates a flow diagram of a second non-limiting embodiment of a method 500 for generating robot path data for robot path(s) to be followed by a robot comprising a coating tool during coating of at least part of the surface of a spatial substrate with at least one coating material. Method 500 may be used for coating at least part of a surface of a spatial substrate with a coating material using a robotic system comprising a robot containing coating tool. The method 500 may contain blocks 102 to 116 as described in relation to FIG. 1 as well as further blocks 502 to 506 described in the following. Method 500 may be performed by the computing device described in relation with FIG. 1, said computing device comprising at least one processor implementing a routine to perform the steps described in relation with the following blocks 502 to 506. Method 500 may be performed by a server device being connected to the computing device performing at least part of the blocks of FIG. 1 via a communication interface. In this setup, the computing device may function as client device and may provide the generated tool path data and provided coating material data to the server device. The server device may then use the received data to perform calculations and may provide the result of the calculations to the client device. This setup may be performed if the computing power of the computing device is not sufficient to perform said calculations.

[0244] In block 502, the routine implementing method 500 may determine whether the amount of each coating material necessary to coat at least part of the surface of each spatial substrate is to be calculated. This determination may be made according to the programming of the routine or may be made, for example, by displaying a graphical user interface prompting the user to select whether said calculation is to be performed or not or by offering a respective button/menu item on the graphical user interface which triggers said calculation upon detection of a user input being indicative of selecting said button/menu item. If the routine determine that the amount of each coating material is to be calculated, it may proceed to block 504. Otherwise, it may end method 500 or may proceed to block 102 of FIG. 1.

[0245] In block 504, the routine implementing method 500 may calculate the amount of each coating material which is necessary to coat at least part of the surface of each spatial substrate based on the tool path data generated in block 112 of FIG. 1 and the coating material data provided in block 102 of FIG. 1.

[0246] In block 506, the routine implementing method 500 may provide the calculated amount of each coating material for each spatial substrate. The calculated amount may be provided to a display device for display within a graphical user interface such that the user can prepare the required amount. The calculated amount along with the provided coating material data may be provided via a communication interface to an automated mixing machine which uses the received data to automatically prepare the calculated amount.

[0247] Performing blocks 504 and 506 may allow the user to obtain information on the amount of coating material that is going to be needed for the coating process and thus allows to use this data for inventory planning or to determine the amount of coating material that needs to be prepared for the coating procedure. The latter avoids preparing more coating material than needed for the painting procedure, allowing to reduce waste and costs associated with waste coating material. Moreover, the determined amount as well as information on the coating material contained in the coating material data, such as the coating material ID, may be provided to an automatic mixing machine which then mixes the determined amount based on the received data. This allows to fully automate the coating process and avoids prepared waste coating material due to mixing errors.

[0248] Blocks 502 to 506 may be performed after block 116 of FIG. 1. Blocks 502 to 506 may be performed after block 114 and prior to block 116 of FIG. 1. This allows to provide the generated robot path data along with the calculated amount of each coating material.

[0249] FIG. 6 shows a computing device 600 that may be used to implement any method set forth in the above-described figures. For instance, the type of computing device 600 shown in FIG. 6 may be used to implement the methods described in relation to FIGS. 1 to 5. In all cases, the computing device 600 represents a physical and tangible processing mechanism.

[0250] The computing device 600 may include one or more hardware processors 602. The hardware processor(s) can include, without limitation, one or more Central Processing Units (CPUs), and/or one or more Graphics Processing Units (GPUs), and/or one or more Application Specific Integrated Circuits (ASICs), etc. More generally, any hardware processor can correspond to a general-purpose processing unit or an application-specific processor unit.

[0251] The computing device 600 may also include computer-readable storage media 604, corresponding to one or more computer-readable media hardware units. The computer-readable storage media 604 retains any kind of information 606, such as computer- or machine-readable instructions, settings, data, etc. Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor 602, cause the computing device 600 to perform a certain function or group of functions. Alternatively or in addition, the computer-executable instructions may configure the computing device 600 to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code. Without limitation, for instance, the computer-readable storage media 604 may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tape, and so on. Any instance of the computer-readable storage media 604 can use any technology for storing and retrieving information. Further, any instance of the computer-readable storage media 604 may represent a fixed or removable component of the computing device 600. Further, any instance of the computer-readable storage media 604 may provide volatile or non-volatile retention of information.

[0252] The computing device 600 may utilize any instance of the computer-readable storage media 604 in different ways. For example, any instance of the computer-readable storage media 604 may represent a hardware memory unit (such as Random Access Memory (RAM)) for storing transient information during execution of a program by the computing device 600, and/or a hardware storage unit (such as a hard disk) for retaining/archiving information on a more permanent basis. In the latter case, the computing device 600 also includes one or more drive mechanisms 608 (such as a hard drive mechanism) for storing and retrieving information from an instance of the computer-readable storage media 604.

[0253] The computing device 600 may perform any of the functions described above when the hardware processor(s) 602 carry out computer-readable instructions stored in any instance of the computer-readable storage media 604. For instance, the computing device 600 may carry out computer-readable instructions to perform each block of the methods described in FIGS. 1 to 5.

[0254] Alternatively, or in addition, the computing device 600 may rely on one or more other hardware logic components 610 to perform operations using a task-specific collection of logic gates. For instance, the hardware logic component(s) 610 may include a fixed configuration of hardware logic gates, e.g., that are created and set at the time of manufacture, and thereafter unalterable. Alternatively, or in addition, the other hardware logic component(s) 610 may include a collection of programmable hardware logic gates that can be set to perform different application-specific tasks. The latter category of devices includes, but is not limited to Programmable Array Logic Devices (PALs), Generic Array Logic Devices (GALs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), etc.

[0255] FIG. 6 generally indicates that hardware logic circuitry 612 includes any combination of the hardware processor(s) 602, the computer-readable storage media 604, and/or the other hardware logic component(s) 610. That is, the computing device 600 can employ any combination of the hardware processor(s) 602 that execute machine-readable instructions provided in the computer-readable storage media 604, and/or one or more other hardware logic component(s) 610 that perform operations using a fixed and/or programmable collection of hardware logic gates. More generally stated, the hardware logic circuitry 612 corresponds to one or more hardware logic components of any type(s) that perform operations based on logic stored in and/or otherwise embodied in the hardware logic component(s).

[0256] In some cases (e.g., in the case in which the computing device 600 represents a user computing device), the computing device 600 also includes an input/output interface 614 for receiving various inputs (via input devices 616), and for providing various outputs (via output devices 618). Illustrative input devices include a keyboard device, a mouse input device, a touchscreen input device, a digitizing pad, one or more static image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, a voice recognition mechanism, any movement detection mechanisms (e.g., accelerometers, gyroscopes, etc.), and so on. One particular output mechanism may include a display device 618 and an associated graphical user interface presentation (GUI) 620. The display device 618 may correspond to a liquid crystal display device, a light-emitting diode display (LED) device, a cathode ray tube device, a projection mechanism, etc. Other output devices include a printer, one or more speakers, a haptic output mechanism, an archival mechanism (for storing output information), and so on. The computing device 600 can also include one or more network interfaces 622 for exchanging data with other devices, such as the robot controller 704 and the databases 708, 710, 712 and 722 of FIG. 7, via one or more communication conduits 624. One or more communication buses 628 communicatively couple the above-described components together.

[0257] The communication conduit(s) 624 may be implemented in any manner, e.g., by a local area computer network, a wide area computer network (e.g., the Internet), point-to-point connections, etc., or any combination thereof. The communication conduit(s) 624 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0258] FIG. 6 shows the computing device 600 as being composed of a discrete collection of separate units. In some cases, the collection of units may correspond to discrete hardware units provided in a computing device chassis having any form factor.

[0259] FIG. 7 illustrates a block diagram of a robotic system 700 for coating at least part of a surface of a spatial substrate with at least one coating material in accordance with implementations of the invention. The robotic system may be used to implement the inventive method, for example method 100 described in relation to FIG. 1, by using the computing apparatus 718 to generate robot path data and to provide the generated robot path data via the robot controller 704 to the robot apparatus 724.

[0260] The robotic system 700 may comprise a computing apparatus 710 and a robot apparatus 702. The computing apparatus 710 may be a computing apparatus 600 as described in FIG. 6 and may comprise a CPU 712 and a storage media 714. The CPU may be a hardware processor or other hardware logic components as described in relation to FIG. 6 above. The storage media 714 may be any computer readable storage media as described in relation to FIG. 6 above. The computer readable storage media may store computer-readable instructions, such as program code, which allow the computing apparatus 710, in particular the CPU 712, to perform the inventive method, such as method 100 described in relation to FIG. 1 and method 500 described in relation to FIG. 5.

[0261] The computing apparatus 710 may be connected via a communication interface to a display device 726, which may receive data from the computing apparatus, such as the 3D model generated in blocks 302 or 304 described in relation with FIG. 3A, generated tool path data, the coverage map calculated in block 312 of FIG. 3A and/or block 324 of FIG. 3B and/or the amount of coating material calculated in block 504 of FIG. 5, and may display the received data with a graphical user interface. For this purpose, either the display device 726 or the computing apparatus 710 may generate a user interface presentation containing the respective data and may provide/display the generated user interface presentation. The display device 726 may be a mobile display device as shown in FIG. 7 or a stationary display device, such as a computer monitor. The display device may comprise a screen, such as an LCD screen, to display the graphical user interface comprising the data generated by computing apparatus 710.

[0262] The computing apparatus 710 may be connected via communication interfaces to different databases 716, 718, 720, 722 and 724. Database 716 may contain spatial substrate data, database 718 may contain coating material data, database 720 may contain coating parameter data, database 722 may contain coating procedure data and database 724 may contain substrate type data as described previously. The data contained in said databases may be interrelated with data allowing computing apparatus 710 to retrieve said data. Part of the data stored in the databases, such as the spatial substrate data, may be stored in the storage media 714 instead of in a database. At least part of the data may be stored in the same database to reduce the number of databases connected to computing apparatus.

[0263] The computing apparatus 710 may be connected to a further computing apparatus being different from computing apparatus 710 (not shown), such as a server device. The server device may be used, for example, to determine the presence of masking materials on the spatial substrate as described in relation to block 104 of FIG. 1, or to perform further blocks of the inventive method, such as method 100 described in relation to FIG. 1.

[0264] The system may further comprise a robot apparatus 702 comprising a robot controller 704 and a robot arm (or movable robot member) 706. The robot apparatus 702 may be connected to the computing apparatus 710 via the robot controller 704 using a communication interface. The robot controller 704 may be configured to receive the robot path data and optionally further commands, such as commands directed to tool changes, and to control, i.e. move, the robot using the received robot path data and further commands. The robot controller may be located outside of a spray booth to avoid a negative influence on the robot controller during the spray operation (see FIG. 9). The robot arm 706 may be any industrial automatic machine with n number of degrees of freedom to which a sensor system or coating tool 708 may be automatically attached. The robot arm 706 may be located inside a spray booth (see FIG. 9 The robot arm may comprise either the sensor system or the coating tool 708. The robot arm 706 may not comprise the sensor system or coating tool 708. In this case, said system or coating tool 708 may be stored in a tool rack (see FIG. 12B). The robot arm 706 may comprise a tool changer which may be configured to attach different coating tools or the sensor system to the robot arm 706. The appropriate coating tool or sensor system 708 may be picked up by the robot arm 706 by providing the robot controller 704 with the location of the tool in the tool rack.

[0265] The sensor system 708 may be configured to generate data of the workspace of the robot as well as to acquire data on the geometry and color of the spatial substrate based on scan path data provided to robot controller 704. The scan path data may be determined as described, for example, in relation to block 210 of FIG. 2. The sensor system may comprise a depth sensor, such as a laser scanner.

[0266] The coating tool 708 may comprise a coating applicator, such as the applicator shown in FIGS. 8A and 8B described later on. The coating tool may further comprise a coating material reservoir comprising a specific coating material as shown in FIGS. 12A and 12B.

[0267] FIG. 8A illustrates the z-axis of a spray applicator 802 of a coating tool in accordance with implementations of the invention. The spray applicator 802 may be an electrostatic spray applicator comprising a nozzle as well as a rotating bell 804. As shown in FIG. 8A, the target distance may be the distance between the end of the nozzle of the spray applicator 802 and the surface of the spatial substrate 806.

[0268] FIG. 8B illustrates the x- and the y-axis of the spray applicator 802 of the coating tool of FIG. 8A. FIG. 8B shows a top view of the spray applicator 802 of FIG. 8A and illustrates the movement of the spray applicator 802 in the vertical and horizontal direction.

[0269] FIG. 9 illustrates a schematic drawing of an example system 900 containing a spray booth 906 and the inventive robotic system, such as robotic system 700 of FIG. 7.

[0270] The system 900 may contain an electrical cabinet 902 housing the electricity necessary for using system 900. The system 900 may further contain a chiller 904 as well as a heat exchanger 916 for the purge air for the robot apparatus, such as the robot arm 706 and/or the coating tool 708 described in relation to FIG. 7.

[0271] The system 900 may further comprise a spray booth 906 comprising spray booth doors 908 to allow placement of the spatial substrate within the spray booth as well as to allow removal of the coated spatial substrate from the spray booth. The spray booth may comprise means for heating (not shown), such that the coating material(s) applied by the robot arm can be dried and/or cured within the spray booth without having to remove the coated spatial substrate and placing the coated spatial substrate within a separate oven.

[0272] The system 900 may further comprise spray booth ventilation 910 to allow ventilation of the spray booth to remove residues of coating material or volatile compounds evaporating from the applied coating material(s) upon drying and/or curing of the resulting coating film.

[0273] The system 900 may further comprise robot controller 912 outside of spray booth 906. This ensures that the robot controller 912 is not negatively influenced by the spray mist as well as the evaporating volatile compounds present within the spray booth during spraying and curing. The robot controller 912 may be connected to robot 914. The robot 914 may comprise a robot arm (see FIG. 11). Suitable robots include robots configured for spraying tasks, such as the FANUC P-50iB/15, which is a 6-axis industrial robot commercially available from Fanuc. The robot may be fixed to a gudel rail system to allow movement of the robot within the spray booth.

[0274] FIG. 10 illustrates a top view of the spray booth 906 and the robot controller 912 of FIG. 9. The robot 914 may be fixed on a bar 1002 which allows movement of the robot along the length of the spray booth 906 using the gudel rail system such that the robot can reach possible locations of the spatial substrate within the spray booth 906. The spray booth may further contain an air outlet 1004 which allows the air provided into the spray booth from the ceiling to flow out through the air outlet. The gantry axis motor configured to move the robot along the spray booth may reside in an explosion proof box 1006 to avoid explosions during spraying solvent-based liquid coating materials containing explosive liquids.

[0275] FIG. 11 illustrates a side view of the part of the spray booth 906 of FIG. 9. A sensor system 706 may be attached to the robot 914. The robot 914 may not have a sensor system or coating tool attached. The robot 914 may have a coating tool (see FIGS. 12A, 12B) attached. The gudel rail system 1102 may be used to move the robot 914 as described previously.

[0276] FIG. 12A illustrates a side view of part of the spray booth 906 of FIG. 9 comprising a tool rack containing a plurality of coating tools, an enclosure housing the sensor system and a gudel rail system 1102.

[0277] The tool rack 1202 may contain a plurality of coating tools 706. Each coating tool may comprise a spray applicator 802 and a coating material reservoir 1206 containing a specific coating material (see FIG. 12B). Each coating tool may be automatically attached to the robot 914 (not shown) using a tool changer.

[0278] The sensor device 706 may be housed inside an enclosure 1204 to avoid contamination of the sensor device during the spraying procedure, which produces spraying mist that might contaminate the sensor device 708. The robot 914 may be configured to open the enclosure and to remove the sensor device from the enclosure 1204 with the use of a tool changer.

[0279] FIG. 12B illustrates a zoom-in on the tool rack and the enclosure housing the sensor system shown in FIG. 12A. The tool rack may contain a plurality of coating tools, such as 4 coating tools. Each coating tool may contain a spray applicator 802 described in relation to FIG. 8 above and a coating material reservoir 1206. The coating material reservoir may contain a specific coating material which may be prepared, for example, by mixing different materials, such as pigment pastes, base varnish, thinner, hardener, in the reservoir or by mixing said materials in a mixing device and filling the resulting coating material into the reservoir. The respective coating tool may be picked up by the robot 914 based on instructions provided by robot controller 912.

[0280] The present disclosure has been described in conjunction with a preferred embodiment as examples as well. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed invention, from the studies of the drawings, this disclosure and the claims. Notably, it is not required that the different steps are performed at a certain place or at one node of a distributed system, i.e. each of the steps may be performed at a different nodes using different equipment/data processing units.

[0281] As used herein determining also includes initiating or causing to determine, generating, querying, accessing, correlating, matching, selecting also includes initiating or causing to generate, access, query, correlating, select and/or match and providing also includes initiating or causing to determine, generate, access, query, correlating, select and/or match, send and/or receive. Initiating or causing to perform an action includes any processing signal that triggers a computing processor to perform the respective action.

[0282] In the claims as well as in the description the word comprising or including does not exclude other elements or steps and the indefinite article a or an does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.