METHOD AND SYSTEM FOR AUTOMATIC ERROR DIAGNOSIS IN A TEST ENVIRONMENT

Abstract

A method for automatic error diagnosis in a test environment is provided. The method comprises the step of providing a plurality of test logs associated with known types of failures, each comprising a set of files. The method further comprises the step of arranging the plurality of test logs in a defect database. Moreover, the method comprises the step of transforming the set of files of the plurality of test logs into vectors adapted to be fed into a machine learning model.

Claims

1. A method for automatic error diagnosis in a test environment comprises the steps of: providing a plurality of test logs associated with known types of failures, each comprising a set of files; arranging the plurality of test logs in a defect database; and transforming the set of files of the plurality of test logs into vectors adapted to be fed into a machine learning model.

2. The method according to claim 1, wherein the method further comprises the step of selecting a part of the files for transformation based on a predefined criterion.

3. The method according to claim 1, wherein the method further comprises the step of performing supervised training of the machine learning model based on the defect database.

4. The method according to claim 3, wherein the supervised training is based on the whole defect database or a part of the defect database.

5. The method according to claim 1, wherein the method further comprises the step of providing a failed test log external to the defect database for error diagnosis.

6. The method according to claim 5, wherein the method further comprises the step of classifying the failed test log by means of the machine learning model with respect to the known types of failures of the defect database.

7. The method according to claim 5, wherein the method further comprises the step of identifying, by means of the machine learning model, if the failed test log does not correspond to any of the known types of failures of the defect database.

8. The method according to claim 1, wherein the method further comprises the step of incorporating additional test logs into the defect database thereby updating the defect database periodically.

9. The method according to claim 8, wherein the method further comprises the step of retraining the machine learning model corresponding to the updated defect database.

10. The method according to claim 1, wherein the set of files belonging to a given test log are in heterogeneous nature.

11. A system for automatic error diagnosis in a test environment comprises: a defect database comprising a plurality of test logs associated with known types of failures, each comprising a set of files; a storage unit configured to store the test logs of the defect database; and a processor connected to the storage unit, configured to transform the set of files of the plurality of test logs into vectors adapted to be fed into a machine learning model.

12. The system according to claim 11, wherein the processor is further configured to select a part of the files for transformation based on a predefined criterion.

13. The system according to claim 11, wherein the processor is further configured to perform supervised training of the machine learning model based on the defect database.

14. The system according to claim 13, wherein the supervised training is based on the whole defect database or a part of the defect database.

15. The system according to claim 11, wherein the system further comprises a user interface and wherein the user interface is configured to input a failed test log external to the defect database for error diagnosis.

16. The system according to claim 15, wherein the processor, by means of the machine learning model, is further configured to classify the failed test log with respect to the known types of failures of the defect database.

17. The system according to claim 15, wherein the processor, by means of the machine learning model, is further configured to identify if the failed test log does not correspond to any of the known types of failures of the defect database.

18. The system according to claim 15, wherein the user interface is further configured to input additional test logs into the defect database thereby updating the defect database periodically.

19. The system according to claim 18, wherein the user interface is further adapted to input the additional test logs in real-time.

20. The system according to claim 18, wherein the processor is further configured to retrain the machine learning model corresponding to the updated defect database.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Exemplary embodiments of the invention are now further explained with respect to the drawings by way of example only, and not for limitation. In the drawings:

[0027] FIG. 1 shows an exemplary embodiment of the method according to the first aspect of the invention;

[0028] FIG. 2 shows an exemplary embodiment of the system according to the second aspect of the invention;

[0029] FIG. 3 shows a first exemplary embodiment of error diagnosis process; and

[0030] FIG. 4 shows a second exemplary embodiment of error diagnosis process.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0031] Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings. However, the following embodiments of the present invention may be variously modified and the range of the present invention is not limited by the following embodiments.

[0032] In FIG. 1, an exemplary embodiment of the method according to the first aspect of the invention is illustrated. In a first step 100, a plurality of test logs are provided that are associated with known types of failures, each comprising a set of files. In a second step 101, the plurality of test logs are arranged in a defect database. In a third step 102, the set of files of the plurality of test logs are transformed into vectors that are adapted to be fed into a machine learning model.

[0033] In FIG. 2, an exemplary embodiment of the system 10 according to the second aspect of the invention is illustrated. The system 10 comprises a defect database 11 comprising a plurality of test logs 12. The defect database 11 is essentially a database of known issues, each of them describing a certain reason for a failed execution of tests. The test logs 12 associated with the defect database 11 are stored in a storage unit or memory 13, either in an ordered or an unordered fashion. The system 10 further comprises a processor 15 and may optionally comprise a display 19. The processor 15 is connected to the storage unit 13 and further to the display 19. Moreover, the system 10 comprises a user interface 17 connected to the storage unit 13 and further respectively to the processor 15 and the display 19.

[0034] In this context, the processor 15 may also be referred to as microcontrollers, microprocessors, microcomputers, etc., which may further be configured in hardware, firmware, software, or a combination thereof. The storage unit 13 can further store the programs required for processing and controlling of the processor 15 and temporarily store input and output information. Therefore, the storage unit 13 may be implemented into a flash memory-type storage medium, a hard disc-type storage medium, a card-type memory, a Random Access Memory (RAM), a Read-Only Memory (ROM), or the like.

[0035] A user may input control commands through the user interface 17 in order to provide control signals for the processor 15 and/or to manipulate the results displayed on the display 19. Test logs can also be inputted through the user interface 17 into the storage unit 13, either individually or collectively. Moreover, test logs external to the database 11, which are required to be analyzed by the processor 15, are also fed through the user interface 17, either mediating via the storage unit 13 or directly into the processor 15. Such inputs can be provided in formats that are known in the art, such as a text file, an XML file and the like. In addition to this or as an alternative, the display 19 may be incorporated with the user interface 17, such as a graphical user interface, where the user can perform the foregoing actions through direct manipulation of graphical elements.

[0036] In FIG. 3, a first exemplary embodiment of error diagnosis process is illustrated in blocks. Here, the defect database 11 and the test logs 12 are illustrated in detail. Each test log 12, (e.g., Log_1, Log_2, Log_3, . . . , Log_N) is associated to a specific failure or defect 21 (e.g., Defect_1, . . . , Defect_N), where the test log 12 may comprise a single file or a set of files 22 (e.g., Log_1_A, Log_1_B, . . . resp. Log_N_A, Log_N_B, . . . ). The processor 15 performs feature extraction 23 on the test logs 12 based on the associated category or type of defects 21.

[0037] For instance, the processor 15 may execute a computer program that maps each file 22 in a single test log 12 to a set of numerical features. In the case a test log 12 comprises a set of files 22, the program combines the feature sets from the set of files 22 into a single feature set and may define the single feature set as log feature set. Each log feature set then can be labeled with the defect 21 associated with the corresponding test log 12. Such labeling of test logs 12 with their associated defects 21 facilitates supervised training of a machine learning model.

[0038] At this point, the processor 15 performs supervised training of the machine learning model. For instance, the processor 15 may execute a computer program that computes a function, which takes a multiplicity of the log feature sets and generates a mapping between a generic feature set and either one or multiple defects 21 in the defect database 11. Additionally, the function may take a multiplicity of the log feature sets and reports that the feature set cannot be mapped to any known defect 21 in the defect database 11.

[0039] Upon formulating the function for training 24, the processor is able to diagnosis an external failed test log 26 corresponding to an unsuccessful test run, which does not belong to the defect database 11. For instance, the processor 15 may execute a computer program that takes the failed test log 26 and by means of the function computed during the supervised training 24, the program maps 25 the failed test log 26 into either one or multiple defects 21 in the defect database 11. The program may then classify 27 the failed test log 26 according to the known defect categories 28 (e.g., Defect_X, Defect_Y, . . . ).

[0040] It is to be noted that additional test logs can be included into the defect database 11, preferably in real-time, in order to enhance the error diagnosis performance or to learn a new defect category. Therefore, the error diagnosis scheme facilitates automatic self-tuning system where the machine learning model can learn from known defects of test logs in order to identify as well as to classify defects for an external test log.

[0041] In FIG. 4, a second exemplary embodiment of error diagnosis process is illustrated in blocks. The process illustrated herein differs from the process of FIG. 3 in that the processor 15 performs prediction 35 on the failed test log 26 in order to reach to a verdict 37 whether the failed test log 26 corresponds to any of the known defects 21. In this regard, the processor 15 may execute a computer program that takes the failed test log 26 and uses the function computed during the supervised training 24 for the prediction 35. Upon reaching a verdict 37, the program may report that the failed test log 26 cannot be mapped to any known defects 21 in the defect database 11. Hence, the processor 15 is able to train the machine learning model based on the defect database 11 in order to predict a known failure of a failed test log 26 or to reach to the conclusion that the failure is not known to the diagnosis process.

[0042] The embodiments of the present invention can be implemented by hardware, software, or any combination thereof. Various embodiments of the present invention may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or the like.

[0043] While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the disclosed embodiments can be made in accordance with the disclosure herein without departing from the spirit or scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the above described embodiments. Rather, the scope of the invention should be defined in accordance with the following claims and their equivalents.

[0044] Although the invention has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. In addition, while a particular feature of the invention may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.