Comparing the performance of multiple versions or branches/paths of an application (e.g., a web service or application) may be conducted within a suitable computing environment. Such an environment may be virtual in nature, cloud-based, or server-based, and is hosted with tools for simultaneously (or nearly simultaneously) executing multiple containers or other code collections with the same or similar operating conditions (e.g., network congestion, resource contention, memory management schemes). By arranging the performance test of different application versions in different sequences executed in parallel in separate containers, fair comparisons of the tested applications will be obtained. Testing sequences may be executed multiple times, and metrics are collected during each execution. Afterward, the results for each metric for each code version are aggregated and displayed to indicate their relative performance quantitatively and/or qualitatively.

A system can identify a file comprising computer-executable instructions, wherein the file has been modified since the file was last transformed into a computer-executable program on which a group of tests was performed. The system can, for respective tests, determine respective line coverage ratios, respective function coverage ratios, and respective branch coverage ratios. The system can select an updated group of tests from the group of tests based on the respective line ratios, the respective function ratios, and the respective branch ratios, the updated group of tests comprising a subgroup of the group of tests. The system can create an updated computer-executable program from the file. The system can test the updated computer-executable program with the updated group of tests.

Various embodiments of the present invention provide methods, apparatuses, systems, computing devices, computing entities, and/or the like for executing efficient and reliable techniques for testing application programming interfaces (APIs) by utilizing at least one of API endpoint modeling data entities and workflow design user interfaces that are generated based at least in part on API endpoint modeling data entities.

Described are methods and systems for predicting and generating impacted scenarios based on a defined set of attributes. The system includes one or more databases. The processors are configured to receive a set of service provider system attributes for a project, generate attribute combinations from the set of service provider system attributes using a machine learning model trained on a reference data model, wherein the reference data model includes multiple test scenarios from the one or more databases, each test scenario associated with a test scenario attribute combination, generate predicted scenarios from the attribute combinations using the machine learning model, determine impacted service provider systems based on the predicted scenarios, determine issues based on each of the predicted scenarios, and generate a complexity score based on the determined impacted service provider systems and the determined issues to determine project viability.

Devices, systems, and methods for upgrading fire panel firmware using machine learning are described herein. In some examples, one or more embodiments include a fire panel comprising a processor and a memory having instructions stored thereon which, when executed by the processor, cause the processor to collect operational data associated with a fire system controlled by the fire panel device over a period of time while the fire panel device utilizes a first firmware version, determine a plurality of test cases based on the operational data, execute the plurality of test cases while the fire panel device utilizes a second firmware version, and provide results of each of the plurality of executed test cases via an interface.

By analyzing a test case in a set of test cases, the test case is classified into a test type. Using a result of analyzing a test execution environment, a flake parameter is set, the flake parameter comprising an execution environment characteristic capable of causing an inconclusive result of execution of the test case. Responsive to determining that the test type maps to the flake parameter, the test case is removed from the set of test cases, the removing resulting in a filtered set of test cases, the determining performed using a predefined set of mappings. The filtered set of test cases is executed in the test execution environment.

A test manager is connected to a hosted version control system containing text files stored in a repository. The test manager receives notification by the version control hosted service of one or more files containing formatted plain text. The formatted plain text includes template language constructs that are pre-processed by the test manager, along with optional defined data, to render manual tests instructions for guiding a human tester to perform operations and observe behavior for a system under test. The user interface is also configured to receive status information from the human tester to be associated with the rendered manual test instructions.

Examples described herein provide a computer-implemented method for identifying regression test failures that includes comparing a base code to a new code to locate an updated aspect of a program. The method further includes inserting debug code into corresponding source files for each of the base code and the new code for the updated aspect. The method further includes building a first image for the base code and a second image for the new code, the first and second images running in respective first and second containers. The method further includes comparing debugging outputs from a regression test of the respective first and second containers to identify a regression test failure. The method further includes implementing a corrective action to correct the regression test failure.

The present disclosure relates to systems, methods, and computer readable media that evaluate performance of a machine learning system in connection with a test dataset. For example, systems disclosed herein may receive a test dataset and identify label information for the test dataset including feature information and ground truth data. The systems disclosed herein can compare the ground truth data and outputs generated by a machine learning system to evaluate performance of the machine learning system with respect to the test dataset. The systems disclosed herein may further generate feature clusters based on failed outputs and corresponding features and generate a number of performance views that illustrate performance of the machine learning system with respect to clustered groupings of the test dataset.

Examples described herein generally relate to performing conformance testing of a computational operation. A reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision can be generated for the computational operation and based on one or more inputs. A hardware result can similarly be created using hardware at a second level of precision. The reference result can be compared to the hardware result to determine a variance value. A conformance result can be output based on whether the variance value is within a threshold range.