An example method of operation may include automatically receiving information from a storage area in response to a signal, which information may include test results for a computer product evaluated by a plurality of test stages. The method may also include retrieving deployment parameters for the computer product, determining whether the test results satisfy the deployment parameters for the computer product, and automatically authorizing deployment of the computer product based on whether the test results satisfy the deployment parameters.

This document describes a framework for measuring and improving the performance of applications, such as distributed applications and web applications. In one aspect, a method includes performing a test on an application. The test includes executing the application on one or more computers and, while executing the application, simulating a set of workload scenarios for which performance of the application is measured during the test. While performing the test, a set of performance metrics that indicate performance of individual components involved in executing the application during the test is obtained. A knowledge graph is queried using the set of performance metrics. The knowledge graph links the individual components to corresponding performance metrics and defines a set of hotspot conditions that are each based on one or more of the corresponding performance metrics for the individual components. A given hotspot condition is detected based on the set of performance metrics.

A hardware monitor arranged to assess performance of a hardware design for an integrated circuit to complete a task. The hardware monitor includes monitoring and counting logic configured to count a number of cycles between start and completion of the symbolic task in the hardware design; and property evaluation logic configured to evaluate one or more formal properties related to the counted number of cycles to assess the performance of the hardware design in completing the symbolic task. The hardware monitor may be used by a formal verification tool to exhaustively verify that the hardware design meets a desired performance goal and/or to exhaustively identify a performance metric (e.g. best case and/or worst case performance) with respect to completion of the task.

A dynamic experimentation evaluation system provides a framework in which a continuous stream of metric data is monitored to establish a causal relationship between changes in a software program and the effect of user-observable behavior. In one aspect, an A/B test is performed continuously on a stream of metric data representing the usage of a control version of software product and the usage of a treatment version of the software product. A sequential probability ratio test (SPRT) is used as the test statistic to determine when to terminate the test and produce results within a specific confidence interval and controlled error rate.

A tool is provided for evaluating contributions of different software developers to a software project, which considers both the structural value and non-structural value of each developer's contributions. Source code for the project is parsed to generate abstract syntax trees (ASTs) representing the code at different times during the project. The tool may perform both a structural value analysis and a non-structural value analysis of the code contributions of each developer. The structural value analysis for each developer may include a call-commit analysis of a call-commit graph generated from the ASTs to analyze function calls and commits attributable to each developer. The non-structural value analysis for each software developer may include identifying and classifying commits attributable to each developer based on impact of such commits on the project. The tool may then calculate a development value score for each developer based on the structural and non-structural metrics for each developer.

A system can determine a group of changed lines in a file comprising computer-executable instructions. The system can, for the respective changed lines, produce a mapping that identifies respective tests that test the respective changed lines. The system can, from the mapping and for the respective tests, determine a merge that identifies respective ones of the respective changed lines that are tested by the respective tests. The system can, from the merge and for the respective tests of the group of tests, determine respective coverage ratios that represent respective ratios between the respective ones of the respective changed lines of the group of changed lines that are tested by the respective tests and respective total changed lines in the file. The system can select an updated group of tests from the group of tests based on the respective ratios.

Systems and methods are described for automated classification of defective code from bug tracking tool data. An example method includes receiving a plurality of datasets representing a plurality of bug reports from a bug tracking application. Each dataset may be generated by vectorizing and clustering a source code associated with a respective bug report represented by the dataset. Each dataset may comprise a plurality of classes. At least one class of each dataset may indicate at least one known bug. For each dataset of the plurality of datasets, a respective supervised feature vector may be generated. Each supervised feature vector may be associated with an index of the at least one class with the at least one known bug. Using the supervised feature vectors, a classification model is trained to detect a new bug presence in a new source code.

There is provided a method for defining an automated test configured, when executed, to test a system under test comprising one or more computer programs being executed on one or more computer devices. The system under test comprises a plurality of operational states, at least one operational state having one or more executable actions associated therewith operable to execute predetermined operations and/or transition the system under test between operational states. The method comprises the steps of: a) defining an executable model of the system under test comprising a plurality of model states, wherein at least some of the model states are representative of operational states of the system under test; and b) defining one or more selectable model actions, each model action being representative of one or more executable actions on the system under test and configured such that one or more preconditions are assignable thereto, wherein each model action is configured to be available to be selected unless any preconditions assigned to the respective model action are not satisfied, wherein the model is configured such that, when executed, a test program is operable to select a sequence of available model actions to define an automated test.

An effective system for verifying safety of an artificial intelligence system includes a feature quantity information accepting unit which accepts feature quantity information that includes values of plural feature quantities, that are assumed as those used in an artificial intelligence system, in each of plural first test data used for a test for verifying safety of the artificial intelligence system; and a judgment unit which judges a first combination, that is a combination that is not included in the plural first test data, in combinations of values that plural feature quantities may take, or a second combination, with it plural correct analysis results that should be derived by the artificial intelligence are associated, in the combinations of the values that the plural feature quantities may take.

According to some embodiments, methods and systems may be associated with enterprise application integration. A formalization platform may facilitate definition of Enterprise Integration Patterns (“EIP”) using a formal representation and defined execution semantics. The formalization platform may also execute model checking to find errors in the formal representation. An implementation platform may receive information from the correctness platform and translate the formal model generated by the correctness platform. The implementation platform may also configure implementation parameters of the translated formal model and simulate the EIP patterns to provide experimental validation. A correctness platform may receive information from the formalization platform and evaluate correctness of the EIP patterns defined via the formalization platform.