By analyzing a test case in a set of test cases, the test case is classified into a test type. Using a result of analyzing a test execution environment, a flake parameter is set, the flake parameter comprising an execution environment characteristic capable of causing an inconclusive result of execution of the test case. Responsive to determining that the test type maps to the flake parameter, the test case is removed from the set of test cases, the removing resulting in a filtered set of test cases, the determining performed using a predefined set of mappings. The filtered set of test cases is executed in the test execution environment.

A test manager is connected to a hosted version control system containing text files stored in a repository. The test manager receives notification by the version control hosted service of one or more files containing formatted plain text. The formatted plain text includes template language constructs that are pre-processed by the test manager, along with optional defined data, to render manual tests instructions for guiding a human tester to perform operations and observe behavior for a system under test. The user interface is also configured to receive status information from the human tester to be associated with the rendered manual test instructions.

Systems and methods for maintaining the safety of a software-based system. One method includes automatically generating a first artifact tree for a hazard for a first version of the system and automatically transforming the first artifact tree into a first augmented tree using a set of heuristics. The method also includes automatically generating a second artifact tree for the hazard for a second version of the system and automatically transforming the second artifact tree for the hazard into a second augmented tree using the set of heuristics. The method further includes automatically comparing the first augmented tree and the second augmented tree to generate a delta view, and automatically generating, based on the delta view, at least one selected from a group consisting of a safety warning for the second version of the software-based system and an actionable recommendation to maintain safety of the second version of the software-based system.

A method of testing a web-based application comprising: at a computing device: receiving a test procedure, the web-based application having an associated web-based application code, the test procedure including at least one testing step comprised of at least an action attribute and an element attribute; for each of the at least one testing steps: identifying an action handler based on the action attribute; generating at least one instruction, by the action handler, and in accordance with a determination that the testing step is an interaction testing step and that there is a value attribute associated with the interaction testing step, modifying the at least one instruction based on the value attribute associated with the interaction testing step; and causing the web-driver to execute the at least one instruction on an identified section of web-based application code corresponding to the element attribute to perform an operation on the web-based application.

An automated testing framework and associated tools for executable applications such as games focus on integration testing, wherein users create data-driven tests by using test modules and configuration data as building blocks. The tools facilitate cooperation between coders and non-technical Quality Assurance (QA) staff in creating automated tests, by simplifying the user interface for configuring tests. Components of the tools simulate user interactions with the application under test, for example, gamepad button presses. The tools are also capable skipping portions of gameplay or other interactive activity and directly jumping into a desired game mode during automated testing, and other functions.

Methods, systems, and devices supporting orchestration for automated performance testing are described. A server may orchestrate performance testing for software applications across multiple different test environments. The server may receive a performance test indicating an application to test and a set of test parameters. The server may determine a local or a non-local test environment for running the performance test. The server may deploy the application to the test environment, where the deploying involves deploying a first component of the performance test to a first test artifact in the test environment and deploying a second component of the performance test different from the first component to a second test artifact in the test environment. The server may execute the performance test to obtain a result set, where the executing involves executing multiple performance test components as well as orchestrating results across multiple test artifacts to obtain the result set.

A system for detecting malicious software, comprising at least one hardware processor adapted to: execute a tested software object in a plurality of computing environments each configured according to a different hardware and software configuration; monitor a plurality of computer actions performed in each of the plurality of computing environments when executing the tested software object; identify at least one difference between the plurality of computer actions performed in a first of the plurality of computing environments and the plurality of computer actions performed in a second of the plurality of computing environments; and instruct a presentation of an indication of the identified at least one difference on a hardware presentation unit.

Examples described herein generally relate to performing conformance testing of a computational operation. A reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision can be generated for the computational operation and based on one or more inputs. A hardware result can similarly be created using hardware at a second level of precision. The reference result can be compared to the hardware result to determine a variance value. A conformance result can be output based on whether the variance value is within a threshold range.

Systems and methods for sourcing participants for a usability study are provided. In some embodiments the systems and methods receive study parameters including the type of study, time-to-field of the study, required number of participants, and required participant attributes. Additionally, a set of business rules for the study are received. These business rules may be received from a client, extrapolated from a service contract with a client for which the study is being performed, or generated based on the monitored outcomes of sourcing of previous studies. Next, panel sources for potential participants and pricing data are queried, and a set of the sources are selected based upon the pricing data. Participants are then received from these sources, which are then fielded in the study and monitored for outcomes.

Provided herein are systems and methods for creating, testing and providing digital guidance for use in an underlying computer application. In one exemplary implementation, a method includes using a browser extension to record steps of digital guidance content as the steps are created by a content author, automatically testing the content with the same browser and making the tested content available for playback. The testing may include receiving an execute testing command, fetching test case data from a server, executing a test case in the browser using the test case data, running tests on the test case in a background tab of the browser, recording passes and or failures of the tests, sending reports of the passes and failures to the server, sending a notification to the content author that the testing results are ready to view, and displaying the testing results in the dashboard.