A method includes receiving, from a computing device, a request for content from a content provider and determining one or more experiment variations of the requested content that the computing device is assigned to based, at least in part, on the request. The method further includes generating, by a processing device, code indicating the determined one or more experiment variations and inserting the code into executable code corresponding to the requested content. The executable code is transmitted to the browser,

A method includes receiving, from a computing device, a request for content hosted by a content provider and determining one or more experiment variations for the requested content that the computing device is assigned to based, at least in part, on the request. The method further includes generating, by a processing device, a hash value of the one or more experiment variations and transmitting the hash value to the computing device and to a caching server.

Off-policy evaluation of a new target policy is performed using historical data gathered based on a previous logging policy to estimate the performance of the target policy. An estimator may be used, wherein either a quality-based estimator or a quality-agnostic estimator is used to weight the difference between an observed reward in the historical data and an estimated reward generated by the target policy. A quality-agnostic estimator may be used to evaluate an importance weight according to a threshold. In such examples, when the importance weight exceeds the threshold, the quality-agnostic estimator clips the importance weight at the threshold, thereby providing an fixed upper bound irrespective of the quality of the reward predictor. In other examples, a quality-based estimator is used, in which an upper bound incorporates the quality of the reward predictor in order to modify an importance weight used by the estimator.

A recommendation system and method access a recommendation bundle pool including multiple recommendation algorithms, each of which is capable of generating one or more recommendations. A first recommendation bundle comprising two or more recommendation algorithms is selected from the pool. Using the first recommendation bundle, recommendations are generated to provide to visitors to a website. When the recommendation system detects a triggering condition for a scaling cycle, the recommendation system applies a scaling mechanism to increase an exploration of additional recommendation bundles from the recommendation bundle pool. Based on the exploration, the recommendation system selects a second recommendation bundle including algorithms selected from the pool. Recommendations are generated using the second recommendation bundle.

The invention enables optimizing performance of a recommendation server. The invention comprises (i) receiving a first set of customer information corresponding to a first set of events recorded in a first time period in which the recommendation server operates in a first configuration state, (ii) generating a first performance evaluation score based on the first set of customer information, (iii) reconfiguring the recommendation server to operate in a second configuration state having a second performance evaluation score associated therewith, and wherein said second performance evaluation score is generated based on a second set of customer information corresponding to a second set of events recorded in a second time period in which the recommendation server operates in the second configuration state and (iv) transmitting to a terminal device, one or more electronic offers selected for transmission to the customer by the recommendation server operating in the second configuration state.

Temporal qualities of events described in electronic documents are verified using a variety of techniques, including human based feedback from contributors near a location of the event in question. The invention can be used in environments such as automated news aggregators, search engines, and other electronic systems which compile information having temporal qualities and which includes timely information on events presented on mobile devices with reduced screen sizes.

Real-time results for events and topics as published in original sourced electronic documents are presented in response to search queries. The invention can be used in environments such as automated news aggregators, search engines, and other electronic systems which compile information having temporal qualities and which includes timely information on events presented on mobile devices with reduced screen sizes.

This application relates to apparatus and methods for identifying and generating audiences. In some examples, a computing device obtains data identifying users and related attributes. The computing device generates static features and dynamic segments based on the data. The computing device obtains an audience definition for an audience, and determines any required static features, dynamic segments, and segments. The computing device determines if any of the required segments need to be generated. To determine whether a segment is to be regenerated, the computing device may determine how much time has elapsed since the last time the segment was generated. The computing device may regenerate segments based on static data less often than segments based on dynamic data. To generate the audience, the computing device applies the audience definition to the required static features, dynamic segments, and segments.

A system and process provide a novel methodology to improve advertising performance by creating and testing new ad copy within advertisements more effectively and efficiently. The system includes an automated technology that produces a virtually unlimited number of predictably high-quality machine-generated advertisements, by combining and recombining the words and phrases (e.g., ad copy) that have proven to be most effective in previous advertisements with new words and phrases that are determined as likely to be effective based on the motivations of the target audience that will be viewing the advertisements. The novel technology is a combined order of specific rules that renders information into a specific format that is then used and applied to create desired results in the form of top performing ad copy and top performing machine-generated advertisements.

This disclosure covers methods, non-transitory computer readable media, and systems that determine a cluster effect representing the impact that a user's digital-content exposure has on other users' conversion actions. The disclosed methods, non-transitory computer readable media, and systems detect the downloads, purchases, or other forms of consumption of a featured item that result from users within the same cluster viewing digital content featuring the item. In some embodiments, the methods, non-transitory computer readable media, and systems apply the cluster effect by, for example, generating a conversion report representing the cluster effect or by providing tools that exploit the cluster effect in distributing digital content.