Described are techniques for A/B testing including a computer-implemented method of identifying, in an A/B testing database, a set of feature values with a statistically significant difference in A/B testing outcomes above a threshold. The method further includes partitioning records in the A/B testing database into a plurality of population strata according to the set of feature values. The method further includes performing A/B testing, and identifying heterogeneous outcomes of the A/B testing for respective strata of the plurality of population strata.

Methods and systems for improved and efficient campaign message management are disclosed. Via an automated process, the system can generate, classify and sort a browsable collection of diverse, high-performing campaign messages, e.g., emails and SMS messages. Such messages can prompt a prospective campaign generator to create quality content for his/her own campaigns. Furthermore, varied and relevant exemplary campaigns can be shown to different users in response to his/her unique needs or expressed interests.

A lift reporting system to perform operations that include: accessing user behavior data associated with one or more machine-learned (ML) models, the ML models associated with identifiers; determining causal conversions associated with the ML models based on the user behavior data, the causal conversions comprising values; performing a comparison between the values that represents the causal conversions; determining a ranking of the ML models based on the comparison; and causing display of a graphical user interface (GUI) that includes a display of identifiers associated with ML models.

Methods, systems, and computer-readable media for user engagement modeling for engagement optimization are disclosed. Based (at least in part) on one or more user engagement models, a user engagement modeling system determines an uplift score for a user of an Internet-accessible service. The uplift score comprises an estimated effect on one or more user engagement metrics of an incentive to interact with the service. The uplift score is determined based (at least in part) on values of the user engagement metric(s) for the user in a treatment group and values of the metric(s) for the user in a control group, in view of propensity score to be in either group. The treatment group is offered the incentive, and the control group is not offered the incentive. The system determines that the user is or is not offered the incentive based at least in part on the uplift score.

An automated method and computer program product are provided for performing multi-channel attribution for conversion events associated with a plurality of consumers. Each consumer has an associated consumer identifier. Each conversion event is associated with a brand or product that has corresponding media advertising electronically delivered on a plurality of different media-based delivery channels to the plurality of consumers via a plurality of media devices associated with respective consumers. Media advertising exposure is electronically detected for each conversion event. Attribution for each of the delivery channels is then electronically determined.

A system provides advertising by using a device discovery process to automatically determine an information about a home network system of a user. When it is determined that a first advertisement has been caused to be presented via a first content providing service or a first media access device, the information is used to automatically prevent a second content providing service or a second media access device from causing a second advertisement to be presented.

A computer-implemented method for placing an advertisement on a web page includes capturing at least a portion of the respective content of a plurality of web pages; transforming the captured content of the web pages to obtain a content description of each respective web page; creating web page groups that contain web pages with content descriptions that are similar to each other to at least a predetermined degree; placing the same advertisement on a plurality of web pages of a first web page group and a second web page group; capturing the number of clicks on the advertisement across the plurality of web pages of the first and second web page groups within a predetermined time period; comparing the number of clicks on the advertisement across the web pages of the first web page group with those of the second web page group; and continuing to use the advertisement on the web pages of the web page group with the higher number of clicks, and discontinuing its use on the web pages of the group with the lower number of clicks.

Apparatuses, methods, and systems for determining winning arms of electronic testing. One method includes obtaining historical data related to the testing, creating a histogram based on the historical data, the histogram including bins and weights, creating a distribution by computing parameters of the distribution from the weights of the histogram, executing the testing, receiving new data collected based on the execution of the test, allocating the new data into same bins as the bins of the histogram of the historical data yielding a new data bin count, computing a posterior distribution comprising updating the distribution using the same bins and the new data bin counts and the parameters of the distribution, inferring corresponding statistics of samplings of a metric distribution, constructing an overall distribution for each arms of the test, and determining a winning arm of the testing.

Systems, methods and media for adaptive real time modeling and scoring are provided. In one example, a system for automatically generating predictive scoring models comprises a trigger component to determine, based on a threshold or trigger, such as a detection of new significant relationships, whether a predictive scoring model is ready for a refresh or regeneration. An automated modeling sufficiency checker receives and transforms user-selectable system input data. The user-selectable system input data may comprise at least one of email, display or social media traffic. An adaptive modeling engine operably connected to the trigger component and modeling sufficiency checker is configured to monitor and identify a change in the input data and, based on an identified change in the input data, automatically refresh or regenerate the scoring model for calculating new lead scores. A refreshed or regenerated predictive scoring model is output.

Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (GPU) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.