METHOD AND SYSTEM FOR INSTRUCTING A USER, PARTICULARLY FOR HANDCRAFTS

Abstract

The present invention relates to a method of instructing a user on performing a task, e.g., a handcraft task, the method comprising: capturing images while the user performs the task, and a user terminal outputting instructions to the user, wherein the output instructions depend on the captured images. The present invention also relates to a corresponding system and use.

Claims

1. A method of instructing a user on performing a handcraft task, the method comprising: capturing images while the user performs the task, and a user terminal outputting instructions to the user, wherein the output instructions depend on the captured images.

2. The method according to claim 1, wherein the method further comprises: a data processing unit processing image data relating to the captured images; and the data processing unit determining a progress of the user in the task based on the processed image data; wherein the output instructions depend on the determined progress of the user.

3. The method according to claim 2, wherein the data processing unit comprises a data storage section comprising a data set corresponding to the task, the data set comprising data portions each corresponding to a step of the task, and wherein the step of determining a progress of the user in the task comprises matching the image data to a data portion in the data set.

4. The method according to claim 3, wherein the step of matching the image data to a data portion in the data set comprises utilization of artificial intelligence and/or a neural network, and/or wherein this step comprises action recognition.

5. The method according to claim 3, wherein in the step of matching image data to a data portion in the data set, image data corresponding to an image captured at a capture time is matched to a data portion in the data set by taking into account, the image data corresponding to the image captured at the capture time, and image data corresponding to an image captured before said capture time.

6. The method according to claim 2, wherein the user terminal comprises the data processing unit.

7. The method according to claim 2, wherein a server external to the user terminal comprises the data processing unit.

8. The method according to claim 2, wherein the images are captured by a camera, wherein the method further comprises the user terminal outputting a positioning instruction instructing the user to position themselves and/or equipment used for the task in a field of view of the camera, and wherein the method further comprises the data processing unit determining whether the image data is sufficient for further processing and wherein the positioning instruction is output if the image data is not sufficient for further processing.

9. The method according to claim 2, wherein the method further comprises the user terminal outputting a status information to the user, wherein the status information depends on the captured images, wherein the method further comprises: the user terminal receiving a dissenting input indicating that the output status is incorrect, and wherein the status information depends on the determined progress of the user.

10. The method according to claim 7, wherein the method further comprises: capturing images while the user performs a defined calibration task; the user terminal receiving a user input relating to parameters of the defined calibration task; and wherein at least some images captured while the user performs the defined calibration task are transferred to the server and are used to train a neural network to recognize an execution of the instruction.

11. The method according claim 3, wherein the method further comprises determining the presence of an error based on the image data and the data set; and the user terminal outputting an error message and/or a correction instruction to the user.

12. The method according to claim 2, wherein the user terminal outputs a plurality of instructions to the user, which output instructions depend on the captured images, and wherein the method further comprises: interrupting outputting the plurality of instructions at an interrupting instruction, and resuming outputting of the plurality of instructions at the interrupting instruction or at an instruction following the interrupting instruction.

13. The method according to claim 3, wherein the data storage section comprises a plurality of data sets, wherein each data set corresponds to a separate task, respectively, and each data set comprises data portions, wherein each data portion corresponds to a step of the respective task, and wherein the method further comprises the user terminal receiving a user input relating to a selection of one of the plurality of data sets.

14. A system configured for carrying out the method according to claim 1, wherein the images are captured by a camera, wherein the system comprises: the user terminal, and the camera, wherein the user terminal comprises the camera.

15. Use of the system according to the preceding claim for carrying out the method according to claim 1.

Description

[0232] The invention will now be described with reference to the accompanying drawings and exemplary embodiments, which are intended to exemplify, but not to limit, the present invention.

[0233] FIG. 1 depicts a system in accordance with an embodiment of the present invention;

[0234] FIG. 2 depicts a flow diagram in accordance with an embodiment of the present invention;

[0235] FIG. 3 depicts a still further flow diagram relating to parts of the flow diagram generally depicted in FIG. 2;

[0236] FIG. 4 depicts another system in accordance with an embodiment of the present invention, as well data structures employed in this embodiment; and

[0237] FIG. 5 depicts a flow diagram corresponding to an embodiment of the present invention.

[0238] FIG. 1 depicts a user 1000 employing an embodiment of the present technology. More particularly, the user 1000 desires to perform a task, such as knitting a garment, e.g., a jumper.

[0239] In the depicted embodiment, there is provided a system 2 comprising a user terminal 10 and a camera 12, which may also be referred to as an image capturing device 12. The camera 12 may be a part of the user terminal 10, i.e., the user terminal 10 may comprise the camera 12. However, in other embodiments, the camera 12 may also be separate from the user terminal 10. The user terminal 10 may be, e.g., a private computer, a laptop, a smartphone, a tablet PC, or a smart speaker. The user terminal 12 comprises an output user interface for communicating information to the user 1000. The output user interface may comprise at least one of a display, a microphone, a headphone, and a speaker. While user terminal 10 and the camera 12, which, in the depicted embodiment is part of the user terminal 10, are depicted to be located on a table where the user 1000 performs their task, other configurations are also possible. That is, the user terminal 10 and/or the camera 12 may also be located at different locations, as long as the camera 12 has a field of view including the task. For example while knitting the preferred position of the camera could be hanging in front of the knitter's body at the height of the knitting task.

[0240] In very simple words, embodiments of the present technology relate to providing interactive instructions to the user 1000. As depicted, the user 1000 performs a task, e.g., a manual task, and more particularly knitting a garment. Very generally, embodiments of the present technology capture pictures of the task, determine a progress of the task and communicate instructions to the user, where the instructions depend on the progress of the task.

[0241] One concrete embodiment of the present technology will now be described in greater detail with reference to FIGS. 2 and 3.

[0242] In a step S1, the user may select a set of instructions. More particularly, the system 2 may comprise a server 50 storing a plurality of sets of instructions. As a mere example, the server 50 may store one instruction set (which may also be referred to as a data set) relating to knitting a jumper, one instruction set relating to knitting a hat, and one instruction set to knitting a jacket.

[0243] The server 50 may communicate with the user terminal 10, e.g., by means of the Internet, as illustrated by the arrows 60, 62, and the user terminal 10 may output (e.g., display) the sets of instructions to the user 1000. Thus, a library of available instruction sets may be displayed to the user 1000. The user 1000 may then select a particular instruction set (e.g., the instruction set knitting a jumper), i.e., they may open a new project with this instruction set. Alternatively, they may also select a saved older project, i.e., a project that they have already worked on.

[0244] Optionally, the user terminal 10 may also output a link to a purchase site and/or recommend new projects, i.e., instruction sets to the user 1000.

[0245] In a step S2, the user 1000 may position equipment for the selected instruction set with respect to the camera 12, i.e., the user 1000 may install a setup for the instruction set. In the described example, the user 1000 may position the knitting equipment, e.g., knitting needles 20 and knitting yarn 22 in front of the camera 12, such that the camera 12 can capture images of the knitting equipment. The camera 12 may be placed at any angle/perspective with respect to the equipment which is convenient and provides a good picture of the working progress.

[0246] More particularly, the sever 50 may communicate instructions relating to the setup to the user terminal 10 and the user terminal 10 may output such instructions to the user 1000.

[0247] In an optional step S3, a computer vision system may be trained to better understand the yarn 22 and the needles 20, e.g., their movement with respect to one another. This may be based on pattern recognition. Generally, the computer vision system may utilize artificial intelligence and/or neural networks to determine, e.g., based on previous results, a progress of the working process. The user terminal 12 may output instructions to the user 1000 instructing the user 1000 to knit a sample, and the user 1000 may knit such a sample. The camera 12 may capture this sample knitting process and the system 2 may determine knitting parameters, such as size, number of stitches, and number of rows. In this step, the system may also be trained to depict the material used (e.g. yarn character or color of the yarn, type of screw), the tools used (e.g. needles, screwdriver, spoon), and/or action performed (e.g. how stitches are made). Thus, the user terminal 10 may then also output all colors of yarn that are needed for the sample knitting process.

[0248] As discussed, the user 1000 may knit the predetermined sample, which may have a pre-determined size, e.g., approximately 10 cm10 cm. The user terminal 10 may then output an instruction asking the user 1000 to enter parameters relating to the knitting of the predetermined sample. For example, the user terminal 10 may request the user 1000 to enter number of stitches and rows for knitting the sample and/or the actual size in cm of the prepared sample, and the user 1000 may input such parameters.

[0249] The user 1000 may input such parameters in a variety of ways by using an input user interface. The input user interface may be a touchscreen, a mouse, a microphone, or a keyboard. In some embodiments, the input user interface may coincide with the output user interface. Thus, the user 1000 may input parameters relating to the sample knitting process.

[0250] Such parameters may be transferred from the user terminal 10 to the server 50, which may comprise a progress recognition unit.

[0251] In an optional step S4, the user terminal 10 may output instructions instructing the user 1000 to input project parameters for the selected instruction set. The project parameters may include, e.g., a size and/or colors of the jumper to be knitted. The user 1000 may then input such project parameters into the user terminal 10 by using the input user interface.

[0252] The instruction set may be adjusted. The adjustment may be based on the parameters based on the sample knitting process and/or on the project parameters. It will be understood that the instructions constituting the instruction set will typically be different, depending on, e.g., a design of a model to be knitted, the size of the jumper to be knitted, chosen colors, and also based on the parameters relating to the sample knitting process (e.g., knitting tension).

[0253] The adjustment of the instruction set may be performed either locally on the user terminal 10, or on the server 50. In the latter case, the project parameters are sent from the user terminal 10 to the sever 50.

[0254] Thus, the instructions of the instruction set may be adjusted, and a linear process of a definite number of instructions may be generated. In case this is performed on the server 50, such a linear instruction set may be communicated from the server 50 to the user terminal 10.

[0255] The user 1000 may then begin with their project, e.g., with their knitting project. This may be accompanied with a step S5, wherein the system 2 and more particularly the user terminal 10 outputs (e.g., displays) an instruction. This instruction may be an initial instruction. In case of a knitting project, this instruction may relate to an instruction what is to be done when knitting the second row. When the initial instruction has been complied with (e.g., when the second row is knitted), the user 1000 may enter a confirmation into the system 2 and more particularly into the user terminal 10.

[0256] Furthermore, the method may also comprise a step S6, which may be referred to as a monitoring and supporting process or monitoring and supporting step. The monitoring and supporting step S6 may comprise different sub-steps that are depicted in FIG. 3.

[0257] In some embodiments, the monitoring and supporting step S6 comprises the sub-step S6a of outputting the user's current progress in the project. This outputting may be performed by the user terminal 10. E.g., in case of a knitting project, the current position (i.e., progress) of the user in the knitting project may be displayed by the user terminal 10. E.g., at the beginning of the project (e.g., a knitting project), the user terminal 10 may output a start point of the project. If a project is resumed after pausing the project, a restart position of the project may be output.

[0258] There may also be provided an option (e.g., for the user 1000) to adjust the progress output by the system 2. E.g., the progress that is output by the user terminal 10 may be erroneous. For example, in a knitting project, it may be possible that the user 1000 started the project under the supervision of the system 2 and that the system 2 captured the user's progress up to a certain point. However, it is possible that the user 1000 continued with the project afterwards, without being monitored by the present system 2. Such further unsupervised progress will then not be captured by the system 2. In this case, the user 1000 may have the option to adjust the progress as output by the system 2.

[0259] Furthermore, the user 1000 may also initiate that the further project (e.g., the knitting project) is captured by the camera 12, and the camera 12 may then capture images of the task (sub-step S6b). Thus, image data is generated by the system 2.

[0260] The generated image data (which may also be referred to as picture data or video data, in case a video is generated) may be transferred to a progress recognition unit. The progress recognition unit may determine the progress of the task with respect to the instruction.

[0261] E.g., in a knitting project, the number of stitches may be determined, the type of stitches (e.g., plain stitches, purl stitches) may be determined, and/or the shape (i.e., color) of the used yarn may be recognized. Further the progress recognition unit may also determine the visibility of the equipment used for the task (e.g., the knitting yarn 22 and the knitting needles 20 in case of a knitting task or knitting project). In case such equipment is not in the field of view of the camera 12, the user terminal 10 may request the user 1000 to reposition the equipment with respect to the camera 12.

[0262] In a further sub-step S6c, a progress of a task within the instruction set may be determined. That is, a location of the progress in the instructions may be determined. For example, this may be performed in a progress tracking system.

[0263] This may be done by means of pattern recognition. In some embodiments, a pattern recognition algorithm may be used. As described, embodiments of the present technology capture images, and more particularly, a series of images of a user performing a task, such as a user knitting a jumper. These images may then be used with a pattern recognition algorithm. The pattern recognition algorithm may, based on an image, determine the progress. Generally, the algorithm may search a given image or images for patterns (e.g., in a knitting process, patterns indicative of: number of stitches, number of rows, type of rows), which patterns may be indicative for the progress in the task. Thus, image contents may be recognized. As a mere example, this may be based on program libraries of openCV.

[0264] In other words, in embodiments of the present technology, there may be provided a database. In the database, there are stored patterns each one corresponding to different steps of a particular task (e.g., a knitting task). For a given image, the presence of such patterns may be determined and it may thus be determined to which step a given image corresponds.

[0265] When knitting there are many identical or similar steps that follow each other. E.g. when executing the instruction knit 10 plain stitches there will be 10 identical sub steps of knitting a plain stitch. Each individual sub step may consist of different parts of action (phases) that needs to be executed. E.g. in order to make a stitch one needle must be pushed through the loop/stitch on the other needle (phase 1), the thread must be wrapped around the needle (phase 2) and the thread must be pulled through the loop/stitch on the second needle (phase 3) and the loop/stitch must be pulled off from the second needle (phase 4).

[0266] Generally, the determination of a task within the instruction set may be done according to different examples.

[0267] In a first example, the determination may be based on an image of the complete project (e.g., of the complete knitting piece prepared so far). Patterns may be recognized on the complete image and thus a progress may be determined.

[0268] In a second example, a working step that only occurs once in the instruction set may be clearly determined. As a mere example, if in a knitting project, there is a stitch where 3 stitches are joined, and this configuration only occurs once in the knitting project, this may be clearly determined. A further example may relate to a handicraft project which comprises the step of folding a sheet of paper in a particular way. Again, if such a step only occurs once in the task, it may be used to determine the progress of the task.

[0269] In a third example, only a portion of the project needs to be captured in an image to obtain information about the progress with respect to an instruction set. For example, if a screw needs to be screwed in at a particular location of a piece of furniture, and the particular location is defined by the shape of the furniture, it may be sufficient to only provide an image of this particular location of the piece of furniture.

[0270] In a fourth example, an image is used allowing the progress in the task to be determined, if the progress at preceding times was determined. For example, in a knitting task, if an image includes stitches and from previous images it is known that the user is currently in row 7, one may determine the progress of the user. This may also be referred to as pattern recognition from one picture based on tracking the progress of the previous work.

[0271] The above examples 1 to 4 may also be referred to as static determination, as a single image is used for the determination of the progress.

[0272] In a still further fifth example, individual phases of the project may be determined (it will be understood that this should also be encompassed by the requirement that a step is determined). For example, in a knitting project, knitting a row may be such a phase. In such a determination, a plurality of images is used (i.e., a section of a video) for the determination, and the plurality of images is used to determine a phase of phases that are ordered one after another. The plurality of images may, e.g., be analyzed by using neuronal networks.

[0273] In one variant of the fifth example, one may also use a determined combination of phases to determine a progress of the user. For example, if it is determined that there is phase of knitting a row followed by changing the color of a yarn, and such a combination of instructions is only present once in the instruction set, the progress of the user may thus be determined.

[0274] In this variant, for example, the system may learn to determine such phases. For example, if an image is matched with a phase, the phase may be considered as executed. If a sequence of certain phases has been executed, it may be determined that a particular step has been executed. As a mere example, in a knitting task, if the phases piercing the needle, getting the yarn, pulling through the yarn, and removing the stitch (which may also be referred to as a loop) from the needle, have been identified, the system may determine that a stitch has been knitted.

[0275] Further, in this variant, the system may also learn to determine such phases in a different manner. E.g., when one or more images with capture times CT-2, CT-1, CT, etc. are matched with a phase, the phase may be considered as executed. Again, a sequence of different such phases may be used for determining a step (e.g., that a stitch has been knitted).

[0276] Still further, in this variant, artificial intelligence, e.g., artificial neural networks (e.g., deep convolutional neural networks) may be used to determine such phases by means of action recognition in the videos. In such versions, a sequence of images (e.g., a section of a video) is used and such phases or actions are thus determined.

[0277] In a further variant of the fifth example, continuously determining the progress based on the above phases may be used. This may be particularly useful for tasks with repetitive steps or phases, such as, knitting, where different working steps are repeated and counted.

[0278] In a sixth example, moving rhythms may be recognized by processing a plurality of images (e.g., a section of a video). Generally, in such images, lines may be generated, their change between (subsequent) images may be determined, and a rhythm, i.e., an interval between corresponding positions may be determined. Subsequent images may be processed according to this rhythm. For example, in a knitting project, the number of stitches that have been knitted can thus be determined. That is, such an example employs a pattern recognition based on a sequence of images.

[0279] The above discussed fifth and sixth example relate to the determination based on a plurality of images, such as sections of a video. They may thus be referred to as dynamic determination,

[0280] In the fifth and sixth example, the phase or rhythm recognition can be based on image recognition which is based on different aspects. For example, in a knitting project, the constellation between the needles and the yarn may be determined (e.g., by convolutional neural networks). Additionally or alternatively, the constellation between the hands and the finger may be determined (e.g., by convolutional neural networks). As an example, different positions and/or degrees of rotations (e.g., between needles, hands, and/or fingers) may be used, e.g., by image processing.

[0281] In do-it-yourself applications, tools, hands, parts, auxiliary devices (e.g., parts of shelf or pins), and their angles and/or rotations may be used. In cooking/baking applications, ingredients, hands, auxiliary devices (e.g., pots, spoons, egg whips) and their angles and/or rotations may be used.

[0282] To further improve the accuracy of such a process, it may also be possible that for determining to which step a particular given image belongs, not only this image is utilized, but also images that have been taken beforehand. That is, the step of determining a progress in the instruction set may not only determine the progress at a given time by a single image shot at this given time, but may also take previous images (or the determination of the progress for such previously captured images) into account. This may render the respective step even more failsafe.

[0283] In some embodiments, fixed algorithms may be used, searching for pre-determined patterns in the images. However, other embodiments may not employ fixed, i.e., pre-determined algorithms, but may employ artificial intelligence tools, and particularly neural networks. In such embodiments, the system may be trained by using numerous training images, and may later, based on the training images, determine a progress of a user in a task. As a mere example, the system may be trained with some few (e.g., 10) and up to 10,000 (e.g., 300) images of a user during a knitting process and the images may be annotated, i.e., the system may be fed with additional information what is depicted on such images. E.g., the additional information may relate to the yarn color used, the number of stitches, and/or the number of rows visible in a picture. The additional information may further relate to: different phases in the process of performing the task/making a new stitch, and particularities of the type of stitch being made. After being trained with such annotated images, the system may then be able to determine a user's progress for a given image or series of images. That is, the recognition of the progress may result from analysis of a single image, but may also result from analysis of a series of images that show different stages of the process. E.g., in a knitting project, only upon completion of all stages of the process the stitch is completed and a corresponding data portion is marked as performed. Also in such embodiments, the system may employ pattern recognition, however, the patterns would then be identified by the system itself.

[0284] In a still further sub-step S6d, the progress determined in the previous step may be output to the user 1000, e.g., by means of the user terminal 10. E.g., in case of a knitting project, there may be given feedback to the user 1000 about where they are in the knitting project at a given moment. As a mere example, the user terminal 10 may output an information such as You are presently at stitch 24 or You are presently at the end of the second row. It will be understood that the progress may be output at specific pre-determined instances, e.g., in a knitting project, at the beginning of a new row, or a pre-determined number of stitches (e.g., 5 to 10 stitches) before a change (such as a change of the yarn color or the type of stitches) occurs. In one embodiment, the system 2 may also output a countdown before such a change occurs (e.g., Number of stitches before change to plain stitches occurs: 5, 4, 3, 2, 1).

[0285] It will be understood that in some instances, the determination of the progress of the task in sub-step S6c, which is output to the user 1000 in sub-step S6d, may be erroneous. Thus, in an optional sub-step S6e, the system 2 and more particularly the user terminal 10 may allow the user 1000 to input a confirmation or a correction relating to the progress that has been determined and output by the system 2. Such an information may then be fed back to the progress tracking system. Such information can also be communicated to the server 50 together with information about the process and may allow the system 2 to be optimized for further usage.

[0286] In a further optional sub-step S6f, the user terminal 10 may output a request for a control view onto the project. E.g. the user terminal 10 may request the user 1000 to position the equipment (e.g., a partly knitted jumper) in a specific way with respect to the camera 12 to thus capture the progress in a pre-determined manner. Thus, e.g., the number of stitches on each needle 20 may be counted.

[0287] The sub-steps described above may all help to determine the progress in a project or task, or to improve such a determination. Once the progress in a project or task (e.g., in a knitting project) has been determined, the system 2 and more particularly the user terminal 10 may output instructions depending on the determined progress of the user 1000 in the task (sub-step S6g). That is, the system 2 may generate such instructions once the location in the project is certain. Mere examples of such an instruction in a knitting project are Another 15 stitches to finish this row, Continue for 6 stitches with the color red and switch to green after that, Now switch to green, Use the green yarn and make 15 stitches. Optionally, the user terminal 10 may also display supporting pictures or video sequences to show to the user 1000 how to correctly perform the next step.

[0288] In a further sub-step S6h, the system 2 may supervise the execution of the instructions, i.e., whether the user 1000 complies with the instructions. More particularly, again, the execution of the task or project is captured by means of the camera 12, and the generated image data may be transferred to a progress recognition unit. It may be compared whether the latest instruction has been complied with, i.e., the step actually performed by the user 1000 and captured by camera 12 may be compared with the instruction output to the user 1000 in sub-step S6g.

[0289] In case the user 1000 complied with the instruction, the system 2 and more particularly the user terminal 10 may output positive feedback to the user 1000, i.e., a green checkmark indicating correct execution of the instructions.

[0290] In case the user 1000 does not (or does not fully) comply with the instruction output in sub-step S6g, the system 2 may recognize this based on the image data and may output (e.g., by means of the user terminal 10) a corrective feedback.

[0291] More particularly, the system 2 may check the execution of task for different characteristics. E.g., in case of a knitting project, the system 2 may, based on the image data, check the characteristics color of yarn, type of stitches, and number of stitches.

[0292] E.g., if the instruction output in sub-step S6g is 15 plain stitches with the red yarn, the user 1000 may not comply with such an instruction in a variety of ways. E.g., they may use another color than red, they may not do plain stitches, or they may perform more (or less) than 15 stitches. The system 2 determines whether any of these characteristics of the instruction has not been complied with. In case the user 1000 has not complied with any such characteristic (i.e., if they have made a mistake), the system 2 can thus identify the type of mistake. Again, in case of a knitting project, the types of possible mistakes include usage of a wrong yarn color or type of stitch or too many or too few stitches (of the correct color or type of stitch).

[0293] In case it is determined that the user 1000 has not complied with an instruction (i.e., has made a mistake), as discussed, the type of this mistake may be determined. Based on the determination of the mistake, the system 2 may generate a corrective instruction. Again with exemplary reference to a knitting project, examples of corrective instructions may include Undo 6 number of stitches or Continue and apply a correction to the next row. The corrective instruction may then be output by the user terminal 10.

[0294] Optionally, the system 2 and more particularly the user terminal 10 may also display pictures of video sequences showing to the user how such a correction is done and/or how to correctly comply with a certain instruction.

[0295] Again, the system 2 constantly captures images of the user 1000 performing the task, and thus also while the user 1000 performs the correction indicated by the corrective instruction. The system 2 may then also determine if the mistake has been corrected. Alternatively or additionally, the user 1000 may also input the completion of the correction into the user terminal 10.

[0296] After a correction has been registered by the system 2 (either by means of the images captured or by means of a user 1000 indicating that a correction has been done), the system 2 may continue to capture images of the task and check whether the instructions are complied with.

[0297] The sub-steps S6a to S6h are (partly optional) sub-steps of the overall step S6 relating to monitoring and supporting the user 1000 while performing their task. This monitoring and supporting step S6 may be stopped or halted in a step S7 (see FIG. 2) in a variety of ways. The user 1000 may input a corresponding stop command into the user terminal 10, e.g., by means of an input user interface.

[0298] Furthermore, as discussed, the system 2 constantly captures images of the user 1000 performing the task, and these images are constantly analyzed. Based on these images, the system 2 may also recognize that the user 1000 has stopped execution of the task (e.g., if the system 2 identifies that no progress has been made in a pre-determined time interval). This may also trigger the monitoring and supporting process to be stopped.

[0299] When the monitoring and supporting step is stopped (S7), the current position in the project, i.e., the current position in the instruction set is captured or saved. Optionally, the user terminal 10 may also output an instruction to position the equipment in a defined manner with regard to the camera 12. This may enable a control view on the project (e.g., the knitting) and the system 2 may thus be able to further check the progress of the project, e.g., to count the stitches on a needle 20 in case of a knitting project.

[0300] It will be understood that the progress of the task may then also be saved, i.e., that the status of the work may be saved by the system 2, for future reference. This may then serve as a restart point when the project is resumed later. Further, based on this, the progress may also be evaluated, and one may reward the user 1000 for the work performed. Further, one may also provide statistics relating to the user and/or to the current project.

[0301] Further details of embodiments of the present invention will now be described with reference to FIG. 4. FIG. 4 again depicts a user terminal, which is here realized as a tablet PC. FIG. 4 again exemplarily relates to a user performing a knitting task, though this is merely exemplary.

[0302] FIG. 4 also depicts a server 50, which may communicate with the user terminal 10, and the communication between the two is exemplarily illustrated by arrows 60, 62. It will be understood that both the server 50 and the user terminal 10 may both send and receive data, e.g., via the internet.

[0303] When the user intends to start with a knitting project, they may use their user terminal 10. The user terminal 10 may comprise a user interface 14. In the present embodiment, the user interface 14 is realized as a touch sensitive display (though it will be understood that other user interfaces, such as microphones and/or a haptic keyboard may also be employed). By means of the user interface 14, the user may input information, e.g., an instruction, into the user terminal 10. If intending to start a new task or project, the user may thus input such an information by means of the user interface 14 into the user terminal 10. More particularly, e.g., the user may use the user terminal 10 to access a website, though this is merely exemplary.

[0304] Thus, a connection may be established between the user terminal 10 and the server 50. The server 50 may comprise a data processing unit. It will be understood that this is merely exemplary, and that in other embodiments the user terminal 10 may comprise the data processing unit. The data processing unit may comprise a data storage section comprising a plurality of data sets IS1 to ISn (see FIG. 4 (a)), which data sets IS1 to ISn may also be referred to as instruction sets. In other words, each instruction set may correspond to an instruction manual. Each such data or instruction set IS1 to ISn relates to a specific task, e.g., IS1 relates to knitting a hat, IS2 relates to knitting a cardigan, IS3 relates to knitting a jumper, and so on. When a connection between the server 50 and the user terminal 10 is established, the instruction sets IS1 to ISn may be displayed to the user by means of the user terminal 10. The user may then select an instruction set, e.g., instruction set IS3 relating to knitting a jumper.

[0305] Each instruction set IS1 to ISn, and thus also instruction set IS3 may comprise a plurality of instructions I1 to Im (see FIG. 4 (b)), which may also be referred to as or may correspond to data portions. An exemplary first instruction I1 could be For the second row, use red yarn and make 10 purl stitches.

[0306] It should be understood that the individual instructions I1 to Im may have different granularities. In the above described example, the first instruction For the second row, use red yarn and make 10 purl stitches relates to one step comprising different steps (e.g., 10 individual stitches). Thus, the described steps I1 to Im may also be composite steps made of a plurality of individual sub steps. It will further be understood that the individual sub steps may also comprise individual sub actions. For example, when the sub step is a single stitch, it may comprise individual sub actions, such as piercing, getting the yarn, pulling through the yarn, and removing a loop from the needles. It will be understood that depending on the concrete project (and expertise of the user), the provided instructions I1 to Im may relate to individual sub actions, sub steps, and/or composite steps. All this should be understood to be encompassed by the present disclosure.

[0307] When the user has selected an instruction set, such as instruction set IS3, in one embodiment, the user terminal 10 may load, e.g., download, this instruction set IS3 with the steps of instructions I1 to Im (however, in some other embodiments, such a download is not necessary, in particular, when there is a continuous connection between the user terminal 10 and the server 50). Independent of whether the instruction set IS3 has been downloaded or not, the user terminal 10 may then have access to it.

[0308] The user terminal 50 may then output the first instruction, e.g., For the second row, use red yarn and make 10 purl stitches, to the user. It may do this in a variety of ways. Generally, the user terminal 50 may comprise an output user interface for communicating such an instruction to the user. The output user interface may be a display, and in the depicted embodiment, may be the described touch sensitive display 14. The user terminal 10 may thus display the instruction I1 to the user. However, in other embodiments, other means may be used to output the instruction I1 to the user. For example, the user terminal 10 may comprise a speaker and may use this as the output user interface. Depending on the task to be performed, this could be advantageous, as the user may not need to look at the user terminal 10 and may thus use their vision for the task to be performed without the user's vision being distracted by the instructions. In any case, the instruction I1 may be output to the user.

[0309] The user may then start to perform their task. In the depicted embodiment, the task may be a knitting task, and thus a knitting yarn 22 and knitting needles 20 are depicted, which may generally be referred to as task equipment. The user may start performing the task by using the task equipment 20, 22.

[0310] While doing so, i.e., while performing the task (e.g., knitting), images of the task are captured by a camera 12, which camera 12 may be part of the user terminal 10. It will be understood that in embodiments of the present technology, it is also ensured that a sufficient portion of the task equipment 20, 22 is in the field of view of the camera 12. Generally, images of the task are captured by the camera 12.

[0311] These images are further analyzed. In some embodiments, this analysis is done locally on the user terminal 10. In other embodiments, this analysis is done externally, i.e., on the server 50.

[0312] Independent of where this analysis is performed, it comprises processing the captured images. In simple terms, the images are processed to determine where in the respective instruction set IS3 the user currently is.

[0313] In particular, embodiments of the present technology determine, based on the captured images, whether the user performing the task is still at a progress corresponding to instruction I1 or not. Again, in the discussed example, the first instruction I1 may be For the second row, use red yarn and make 10 purl stitches.

[0314] The presently described method may thus process the captured images to determine whether the user is still knitting the second row and how many stitches they have performed. This may be done by using pattern recognition, e.g., a pattern recognition algorithm. Thus, patterns in the captured images may be recognized to determine the progress of the user. As a mere example, algorithms of the openCV libraries may be employed. Implementations may use some part of programs or use the technology of Yolo, Darknet, tinyYolo, CoreML, Python, and/or Caffe toolbox.

[0315] Thus, the user's progress may be determined. As long as it is determined that he user's progress corresponds to the first instruction, the first instruction may be output to the user.

[0316] However, once it is determined that the user has completed the step corresponding to the first instruction (or closely before completion of the first instruction, e.g., 3 to 5 stitches before the end of the second row), the present technology may determine this and change the instruction output to the user accordingly.

[0317] E.g., the second instruction I2 may be For the third row, use black yarn and make 10 plain stitches. As a mere example, this instruction may be output to the user when they have completed the second row (or closely before such completion). In the latter case, it is also possible that the present technology more particularly instructs the user, e.g., by providing a countdown before the next instruction I2 applies.

[0318] The above rationale may then apply to the whole process, project, or task the user is performing. That is, in very general words, embodiments of the present technology monitor the performed task, i.e., capture images thereof, and automatically determine the progress of the task. Depending on the determined progress of the task, the instructions that are output to the user are given. More particularly, only the instruction that presently applies may be output to the user. This may be advantageous, as the user may focus their attention on the actual task to be performed, without their attention being captured by determining their progress in an instruction.

[0319] It will be understood that embodiments of the present technology may also allow the user to pause the described method. E.g., when a user wants to interrupt their project, e.g., their knitting project, they can input this to the user terminal 10, and the project may then be halted, and restarted at the respective point at a later time.

[0320] FIG. 5 is a very conceptual schematic representation of an embodiment of the present invention. Very generally, the embodiment again relates to a method for identification of progress while doing a task, e.g., handicrafts (such as knitting, crocheting and the like).

[0321] Again, the task is identified by respective equipment, which is here realized as knitting needles 20 and knitting yarn 22. The task may be the generation of a garment 24, e.g., a jumper. Again, a camera 12, which may be part of a user terminal 10, captures images of the task and thus generates image data 70. The image data 70 is sent to a data processing system 72. As discussed, such a processing system 72 may be part of a user terminal 10 and/or of an external server 50. The processing system 72 may sent data to a control and navigation system 74, which also has access to an instruction 76 (e.g., a set of instructions stored in a data base). Further, the control and navigation system 74 may also receive additional user input 78, e.g., input registering corrections of the user. All these inputs may be processed in the control and navigation system 74 to generate an output 80, which may by an output including user instructions and/or status information on the task.

[0322] Again, this embodiment generally relates to a method for identification of progress while doing a task, such as a handicraft task (e. g. knitting, crocheting and the like) based on image or video data 70, which are recorded by a camera 12 (e.g., of a user terminal) and the continuous instruction of the person doing handicrafts as regards the subsequent steps in knitting or crocheting depending on his/her work progress. The image or video data 70 may be used to record a quantity of work steps such as e.g. a number of stitches on a knitting needle 20, or in order to record an individual work step, such as e.g. by tracking the movements of the needles 20 and the yarn 22 in creating a new stitch. Instructions 76 for creation of handicrafts (such as knitting or crocheting work) represent the description of the method to be tracked by the persons doing handicrafts. These instructions 76 are typically provided by the system and transferred to the control and navigation system 74, which may also be referred to as the processing unit 74, to thus permit the work progress to be located in the instructions 76. Recording of the work steps with camera 12 and by using the processing system 72 are utilized to position/track the progress of the work within the instructions. The control and navigation system 74 records the work progress as it occurs and positions it in the instructions.

[0323] The user is informed about their progress via visual display and/or auditive announcements, i.e., by the output 80.

[0324] It will be understood that any of the described embodiments may malfunction, i.e., not correctly locate the user's progress in the task. For example, the user may be outside the range of vision of the camera 12 and/or work may be performed when the system is not operating (e.g., outside of the operating hours of the image recording unit). In such instances, the user may update or adjust the progress status on a user interface of the user terminal by means of an additional user input 78.

[0325] In line with the progress or positioning in the work, the next upcoming work steps are indicated and/or announced by output 80 to the user based on the instructions. Such announcement might be, for example: now knit another 4 stitches in red and then knit the following 15 stitches in white or knit another 2 plain stitches, then increase with one purl stitch.

[0326] Again, it is also possible to display images or video sequences which illustrate the precise course of action visually to the user.

[0327] While in the above, preferred embodiments have been described with reference to the accompanying drawings, the skilled person will understand that these embodiments were provided for illustrative purpose only and should by no means be construed to limit the scope of the present invention, which is defined by the claims.

[0328] Furthermore, while in the above, different aspects were discussed with reference to different Figures, it should be understood that these aspects can also be readily combined with one another. E.g., the user terminal 10 depicted in FIGS. 1 and 4 may be employed in the method discussed in conjunction with FIGS. 2 and 3. Further, with regard to FIGS. 2 and 3, a particular process comprising a plurality of steps and sub steps has been described. It will be understood that these steps and sub steps may also be employed with the system and method generally described with reference to FIGS. 1, 4 and/or 5.

[0329] It will be understood that the present technology may employ computer vision (CV) technologies. CV technologies relate to methods and tools that are directed at gaining high level understanding of digital images or video. Understanding in this context means the transformation of visual images into descriptions of the world that can interface with other thought processes and elicit appropriate action. Computer vision is concerned with the theory behind artificial systems that extract information from images and with the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images. Generally, the realisation of extraction and analysis of the visual information may rely on systematic explicit algorithms and/or may use Artificial Neural Networks (ANN) and particularly the convolutional neural networks (CNN).

[0330] Generally, in order to determine the execution of a given step in a task the present technology may also comprise methods developed in the field of action recognition in order to determine the full execution of the given step, which may be different from the elicitation of the execution by identifying the phases and their completion.

[0331] While in the above, particular embodiments relating to instructing a user performing a task have been described, it should be understood that the present technology may also be used to capture images of a user performing a task (which may be unknown to the system) and to thus generate a manual, i.e., an instruction set, based on this.

[0332] Whenever a relative term, such as about, substantially or approximately is used in this specification, such a term should also be construed to also include the exact term. That is, e.g., substantially straight should be construed to also include (exactly) straight.

[0333] Whenever steps were recited in the above or also in the appended claims, it should be noted that the order in which the steps are recited in this text may be accidental. That is, unless otherwise specified or unless clear to the skilled person, the order in which steps are recited may be accidental. That is, when the present document states, e.g., that a method comprises steps (A) and (B), this does not necessarily mean that step (A) precedes step (B), but it is also possible that step (A) is performed (at least partly) simultaneously with step (B) or that step (B) precedes step (A). Furthermore, when a step (X) is said to precede another step (Z), this does not imply that there is no step between steps (X) and (Z). That is, step (X) preceding step (Z) encompasses the situation that step (X) is performed directly before step (Z), but also the situation that (X) is performed before one or more steps (Y1), . . . , followed by step (Z). Corresponding considerations apply when terms like after or before are used.