GENERATING ARTIFICIAL INTELLIGENCE SUBJECTS AND MEDIA STIMULI TO SIMULATE AND PREDICT HUMAN RESPONSES
20260004019 ยท 2026-01-01
Inventors
- Leonid Yeykelis (Mountain View, CA, US)
- James Cummings (Boston, MA, US)
- Byron Reeves (Palo Alto, CA, US)
Cpc classification
International classification
Abstract
According to various embodiments described herein, mechanisms are provided for employing artificial intelligence (AI), such as in the form of large language models, to simulate responses to stimuli. A study may be created, a set of AI subjects having specified characteristics may be generated, and stimuli may be generated and presented to such AI subjects. Responses may be recorded and analyzed, and reports may be generated based on such analysis. The described system and method provide mechanisms for making research and/or product decisions based on observed responses by the AI subjects to stimuli.
Claims
1. A computer-implemented method for simulating responses to media stimuli in order to compare effectiveness of different versions of the media, comprising: at an input device, receiving a first set of parameters for generating artificial intelligence (AI) subjects to be run on at least one hardware processor to simulate human subjects, wherein the first set of parameters defines a set of characteristics for the AI subjects; at the at least one hardware processor, generating a plurality of AI subjects according to the first set of parameters, to simulate human subjects; at a storage device, storing data defining the plurality of AI subjects; at the at least one hardware processor, running the generated AI subjects; at the input device, receiving a second set of parameters for stimuli to be presented to the generated AI subjects, the stimuli representing different versions of the media; at the at least one hardware processor, generating stimuli according to the second set of parameters; at the at least one hardware processor, presenting each generated stimulus to at least a subset of the generated AI subjects; at the at least one hardware processor, simulating human responses to the stimuli to predict future responses to the different versions of the media, based on the responses of the generated AI subjects to the stimuli; at the at least one hardware processor, recording responses of the generated AI subjects to the stimuli, at the at least one hardware processor, automatically comparing effectiveness of the different versions of the media, wherein comparing the effectiveness of the different versions comprises determining, from the responses of the generated AI subjects to the stimuli, the effectiveness of the different versions of the media according to at least one metric; at the at least one hardware processor, automatically selecting among the different versions of the media based on the comparison; at an output device, outputting a report based on the recorded responses and the comparison; and at the storage device, storing the recorded responses and the results of the selection.
2. The method of claim 1, wherein the received first set of parameters defines a distribution of demographic characteristics for the AI subjects.
3. The method of claim 1, wherein the stimuli comprises at least one selected from the group consisting of: text; at least one image; and video content; and wherein presenting the generated stimuli to the generated AI subjects comprises: translating the stimuli to a prompt grammar; creating a prompt for presentation to the generated AI subjects; and presenting the prompt to the generated AI subjects.
4. The method of claim 1, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: assigning a condition to each generated AI subject; and determining specific stimuli to be presented to each generated AI subject based on its assigned condition.
5. The method of claim 1, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: creating a plurality of measures, each measure comprising at least one question to be asked of a generated AI subject; and prompting the generated AI subject to respond to the at least one question.
6. (canceled)
7. The method of claim 1, wherein: presenting each generated stimulus to at least a subset of the generated AI subjects comprises sending at least one prompt having a type and representing the generated stimuli to a plurality of large language model instances, each executing code representing one of the generated AI subjects; and each large language model instance is adapted to automatically detect the type of each prompt.
8. The method of claim 1, wherein: comparing the effectiveness of the different versions of the media comprises, at the at least one hardware processor, running a statistical analysis of the recorded responses of the generated AI subjects to the stimuli; and the report is based on the statistical analysis.
9. The method of claim 8, wherein running the statistical analysis comprises automatically determining which of a plurality of techniques to use for the statistical analysis.
10. The method of claim 8, further comprising automatically generating additional stimulus for further research, based on the results of the statistical analysis.
11. A non-transitory computer-readable medium for simulating responses to media stimuli in order to compare effectiveness of different versions of the media, comprising instructions stored thereon, that when performed by at least one hardware processor, perform the steps of: causing an input device to receive a first set of parameters for generating artificial intelligence (AI) subjects to be run on at least one hardware processor to simulate human subjects, wherein the first set of parameters defines a set of characteristics for the AI subjects; generating a plurality of AI subjects according to the first set of parameters, to simulate human subjects; causing a storage device to store data defining the plurality of AI subjects; running the generated AI subjects; causing the input device to receive a second set of parameters for stimuli to be presented to the generated AI subjects, the stimuli representing different versions of the media; generating stimuli according to the second set of parameters; presenting each generated stimulus to at least a subset of the generated AI subjects; simulating human responses to the stimuli to predict future responses to the different versions of the media, based on the responses of the generated AI subjects to the stimuli; automatically comparing effectiveness of the different versions of the media, wherein comparing the effectiveness of the different versions comprises determining, from the responses of the generated AI subjects to the stimuli, the effectiveness of the different versions of the media according to at least one metric; automatically selecting among the different versions of the media based on the comparison; recording responses of the generated AI subjects to the stimuli; causing an output device to output a report based on the recorded responses and the comparison; and causing the storage device to store the recorded responses and the results of the selection.
12. The non-transitory computer-readable medium of claim 11, wherein the received first set of parameters defines a distribution of demographic characteristics for the AI subjects.
13. The non-transitory computer-readable medium of claim 11, wherein the stimuli comprises at least one selected from the group consisting of: text; at least one image; and video content; and wherein presenting the generated stimuli to the generated AI subjects comprises: translating the stimuli to a prompt grammar; creating a prompt for presentation to the generated AI subjects; and presenting the prompt to the generated AI subjects.
14. The non-transitory computer-readable medium of claim 11, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: assigning a condition to each generated AI subject; and determining specific stimuli to be presented to each generated AI subject based on its assigned condition.
15. The non-transitory computer-readable medium of claim 11, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: creating a plurality of measures, each measure comprising at least one question to be asked of a generated AI subject; and prompting the generated AI subject to respond to the at least one question.
16. (canceled)
17. The non-transitory computer-readable medium of claim 11, wherein: presenting each generated stimulus to at least a subset of the generated AI subjects comprises sending at least one prompt having a type and representing the generated stimuli to a plurality of large language model instances, each executing code representing one of the generated AI subjects; and each large language model instance is adapted to automatically detect the type of each prompt.
18. The non-transitory computer-readable medium of claim 11, wherein: comparing the effectiveness of the different versions of the media comprises running a statistical analysis of the recorded responses of the generated AI subjects to the stimuli; and the report is based on the statistical analysis.
19. The non-transitory computer-readable medium of claim 18, wherein running the statistical analysis comprises automatically determining which of a plurality of techniques to use for the statistical analysis.
20. The non-transitory computer-readable medium of claim 18, further comprising instructions stored thereon, that when performed by the hardware processor, perform the step of automatically generating additional stimulus for further research, based on the results of the statistical analysis.
21. A system for simulating responses to media stimuli in order to compare effectiveness of different versions of the media, comprising: an input device, configured to receive a first set of parameters for generating artificial intelligence (AI) subjects to be run on at least one hardware processor to simulate human subjects, wherein the first set of parameters defines a set of characteristics for the AI subjects; at least one hardware processor, each communicatively coupled to the input device, configured to generate a plurality of AI subjects according to the first set of parameters, to simulate human subjects; a storage device, communicatively coupled to the at least one hardware processor, configured to store data defining the plurality of AI subjects; and an output device, communicatively coupled to the at least one hardware processor; wherein: the input device is further configured to receive a second set of parameters for stimuli to be presented to the generated AI subjects, the stimuli representing different versions of the media; the at least one hardware processor is further configured to: run the generated AI subjects; generate stimuli according to the second set of parameters; present each generated stimulus to at least a subset of the generated AI subjects; simulate human responses to the stimuli to predict future responses to the different versions of the media, based on the responses of the generated AI subjects to the stimuli; automatically compare effectiveness of the different versions of the media, wherein comparing the effectiveness of the different versions comprises determining, from the responses of the generated AI subjects to the stimuli, the effectiveness of the different versions of the media according to at least one metric; automatically select among the different versions of the media based on the comparison; and record responses of the generated AI subjects to the stimuli; the an output device is configured to output a report based on the recorded responses and the comparison; and the storage device is further configured to store the recorded responses and the results of the selection.
22. The system of claim 21, wherein the received first set of parameters defines a distribution of demographic characteristics for the AI subjects.
23. The system of claim 21, wherein the stimuli comprises at least one selected from the group consisting of: text; at least one image; and video content; and wherein presenting the generated stimuli to the generated AI subjects comprises: translating the stimuli to a prompt grammar; creating a prompt for presentation to the generated AI subjects; and presenting the prompt to the generated AI subjects.
24. The system of claim 21, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: assigning a condition to each generated AI subject; and determining specific stimuli to be presented to each AI subject based on its assigned condition.
25. The system of claim 21, wherein presenting each generated stimulus to at least a subset of the generated AI subjects comprises: creating a plurality of measures, each measure comprising at least one question to be asked of a generated AI subject; and prompting the generated AI subject to respond to the at least one question.
26. (canceled)
27. The system of claim 21, wherein: presenting each generated stimulus to at least a subset of the generated AI subjects comprises sending at least one prompt having a type and representing the generated stimuli to a plurality of large language model instances, each executing code representing one of the generated AI subjects; and each large language model instance is adapted to automatically detect the type of each prompt.
28. The system of claim 21, wherein: comparing the effectiveness of the different versions of the media comprises running a statistical analysis of the recorded responses of the generated AI subjects to the stimuli; and the report is based on the statistical analysis.
29. The system of claim 28, wherein running the statistical analysis comprises automatically determining which of a plurality of techniques to use for the statistical analysis.
30. The system of claim 28, wherein the hardware processor is further configured to automatically generate additional stimulus for further research, based on the results of the statistical analysis.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings, together with the description, illustrate several embodiments. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit scope.
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
DETAILED DESCRIPTION OF THE EMBODIMENTS
[0038] The systems and methods set forth herein may be applied in many contexts in which it may be useful to simulate and/or predict human responses to media stimuli, such as to determine potential effectiveness of marketing campaigns, assess usability of software and other products, identify potential problems, formulate research questions instrumental to testing new media psychology theory, and/or the like. Such techniques may be useful to provide feedback associated with different versions of products, media, and/or other items, and may be used to replace and/or enhance conventional mechanisms for testing such items. Accordingly, one context for the techniques presented herein is to increase the robustness of testing methods.
[0039] For illustrative purposes, the systems and methods are described in connection with a real-time experiment that may be performed in a turn-based chat interaction. AI subjects (also referred to as simulated humans, simulated participants, AI participants, or virtual personas) may be generated and enlisted to participate in conversations that simulate responses to potential media messages, wherein each turn may take into account a user's message and message history up to that point.
[0040] However, the described techniques may also be used in other contexts, and are not limited to the specific application(s) set forth herein. For example, the described techniques may be applied to simulate human responses to stimuli in other contexts, and are not necessarily limited to conversations in a chat interaction. As another example, the described techniques may be applied to a longitudinal study such as one testing the effectiveness of an advertising campaign over several months.
[0041] In addition, for illustrative purposes, systems and methods are described in connection with studies involving individuals and their responses to stimuli. However, the described techniques may also be applied in contexts where responses of groups or other entities are considered, such as for example, organizations, families, schools, governments, countries, and/or companies.
[0042] In addition, the particular hardware arrangements depicted and described herein are simplified examples for illustrative purposes.
[0043] In some embodiments, one or more hardware and/or software components, as shown and described below in connection with
[0044] Further, the functions and/or method steps set forth herein may be carried out by software running on one or more of device(s) 101, client device(s) 108, server(s) 110, and/or other components. This software may optionally be multi-function software that is used to retrieve, store, manipulate, and/or otherwise use data stored in data storage devices and/or to carry out one or more other functions.
Definitions and Concepts
[0045] For illustrative purposes and for ease of explanation, the following definitions and concepts are used herein: [0046] A customer, user, or end user, such as user 100 referenced herein, may be an individual person, or may be an enterprise, company, family, representative, and/or group that may optionally include one or more individuals. [0047] A data store, such as data store 106 referenced herein, may be any device capable of digital data storage, including any known hardware for nonvolatile and/or volatile data storage. A collection of data stores may form a data storage system that can be accessed by multiple customers. [0048] A computing device or device, such as client device 101 or 108, is any electronic device capable of digital data processing. In at least one embodiment, the device may communicate with a server such as server 110, provide output to a customer, and accept input from a customer. [0049] A server, such as server 110, may be a computing device that is configured to run software for back-end processing, and may also provide data storage, either via a local data store, or via connection to a remote data store. [0050] An AI subject (also referred to as a simulated human, AI participant, or virtual persona) may be a software-based entity capable of receiving stimuli and generating a response to such stimuli. In at least one embodiment, such entities may be implemented using a large language model (LLM) and/or other machine learning technique(s). In other embodiments, such entities may be implemented via other approaches. Notably, in at least one embodiment, AI subjects may be capable of processing multi-modal stimuli, such as for example images, sounds, and text from videos, VR sims, and/or the like.
System Architecture
[0051] According to various embodiments, the systems and methods described herein may be implemented on any electronic device or set of interconnected electronic devices, each equipped to receive, store, and present information. Each electronic device may be, for example, a server, desktop computer, laptop computer, smartphone, tablet computer, smart watch or other wearable device, and/or the like. As described herein, some devices may be designated as client devices, which are generally operated by end users. Other devices may be designated as servers, which generally conduct back-end operations and communicate with client devices (and/or with other servers) via a communications network such as the Internet. In at least one embodiment, the techniques described herein may be implemented in a cloud computing environment using techniques that are known to those of skill in the art.
[0052] In addition, one skilled in the art will recognize that the techniques described herein may be implemented in other contexts, and indeed in any suitable device, set of devices, or system capable of interfacing with existing enterprise data storage systems. Accordingly, the following description is intended to illustrate various embodiments by way of example, rather than to limit scope.
[0053] Referring now to
[0054] In at least one embodiment, device 101 includes a number of hardware components that are well known to those skilled in the art. Input device 102 can be any element that receives input from user 100, including, for example, a keyboard, mouse, stylus, touch-sensitive screen (touchscreen), touchpad, trackball, accelerometer, microphone, or the like. Input can be provided via any suitable mode, including for example, one or more of: pointing, tapping, typing, dragging, and/or speech. In at least one embodiment, input device 102 may be implemented as a device that can unobtrusively assess a reaction without intentional input from user 100, for example by monitoring psychophysiological changes or by capturing behavioral choices.
[0055] In at least one embodiment, input device 102 can be omitted or functionally combined with one or more other components.
[0056] Data store 106 can be any magnetic, optical, or electronic storage device for data in digital form; examples include flash memory, magnetic hard drive, CD-ROM, DVD-ROM, or the like. In at least one embodiment, data store 106 may store information that can be utilized and/or displayed according to the techniques described below. Data store 106 may be implemented in a database or using any other suitable arrangement. In another embodiment, data store 106 can be stored elsewhere, and data from data store 106 can be retrieved by device 101 when needed for processing and/or presentation to user 100. Data store 106 may store one or more data sets, which may be used for a variety of purposes and may include a wide variety of files, metadata, and/or other data.
[0057] In at least one embodiment, data store 106 may store data for performing various tasks and operations in connection with the functionality described herein, including for example creating media stimuli, generating and running AI subjects, recording test results, and/or the like. In at least one embodiment, some or all of such data can be stored at another location, remote from device 101, and device 101 can access such data over a network, via any suitable communications protocol.
[0058] In at least one embodiment, data store 106 may be organized in a file system, using well-known storage architectures and data structures, such as relational databases. Examples include Oracle, MySQL, and PostgreSQL. Appropriate indexing can be provided to associate data elements in data store 106 with each other. In at least one embodiment, data store 106 may be implemented using cloud-based storage architectures such as NetApp (available from NetApp, Inc. of Sunnyvale, California) and/or Amazon Simple Storage Service (Amazon S3) (available from Amazon.com of Seattle, Washington).
[0059] Data store 106 can be local or remote with respect to the other components of device 101. In at least one embodiment, device 101 is configured to retrieve data from a remote data storage device when needed. Such communication between device 101 and other components can take place wirelessly, by Ethernet connection, via a computing network such as the Internet, via a cellular network, or by any other appropriate communication systems.
[0060] In at least one embodiment, data store 106 is detachable in the form of a CD-ROM, DVD, flash drive, USB hard drive, or the like. Information can be entered from a source outside of device 101 into data store 106 that is detachable, and later displayed after data store 106 is connected to device 101. In another embodiment, data store 106 is fixed within device 101.
[0061] In at least one embodiment, data store 106 may be organized into one or more well-ordered data sets, with one or more data entries in each set. Data store 106, however, can have any suitable structure. Accordingly, the particular organization of data store 106 need not resemble the form in which information from data store 106 is displayed to user 100 on display screen 103. In at least one embodiment, an identifying label is also stored along with each data entry, to be displayed along with each data entry.
[0062] Display screen 103 can be any element that displays information such as text and/or graphical elements. In particular, display screen 103 may present a user interface for entering, viewing, configuring, selecting, editing, downloading, and/or otherwise interacting with data as described herein, including media stimuli, AI subjects, test results, and/or the like. In at least one embodiment where only some of the desired output is presented at a time, a dynamic control, such as a scrolling mechanism, may be available via input device 102 to change which information is currently displayed, and/or to alter the manner in which the information is displayed. In at least one embodiment, display screen 103 can be omitted or functionally combined with one or more other components.
[0063] Processor 104 can be a conventional microprocessor for performing operations on data under the direction of software, according to well-known techniques. Memory 105 can be random-access memory, having a structure and architecture as are known in the art, for use by processor 104 in the course of running software.
[0064] Communication device 107 may communicate with other computing devices via any known wired and/or wireless protocol(s). For example, communication device 107 may be a network interface card (NIC) capable of Ethernet communications and/or a wireless networking card capable of communicating wirelessly over any of the 802.11 standards. Communication device 107 may be capable of transmitting and/or receiving signals to transfer data and/or initiate various processes within and/or outside device 101.
[0065] Referring now to
[0066] Client device 108 can be any electronic device incorporating input device 102 and/or display screen 103, such as a desktop computer, laptop computer, personal digital assistant (PDA), cellular telephone, smartphone, music player, handheld computer, tablet computer, kiosk, game system, wearable device, or the like. Any suitable type of communications network 109, such as the Internet, can be used as the mechanism for transmitting data between client device 108 and server 110, according to any suitable protocols and techniques. In addition to the Internet, other examples include cellular telephone networks, EDGE, 3G, 4G, 5G, long term evolution (LTE), Session Initiation Protocol (SIP), Short Message Peer-to-Peer protocol (SMPP), SS7, Wi-Fi, Bluetooth, ZigBee, Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (SHTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), and/or the like, and/or any combination thereof. In at least one embodiment, client device 108 transmits requests for data via communications network 109, and receives responses from server 110 containing the requested data. Such requests may be sent via HTTP as remote procedure calls or the like.
[0067] In one implementation, server 110 is responsible for data storage and processing, and may incorporate data store 106. Server 110 may include additional components as needed for retrieving data from data store 106 in response to requests from client device 108.
[0068] As described above in connection with
[0069] In addition to or in the alternative to the foregoing, data may also be stored in data store 106 that is part of client device 108. In some embodiments, such data may include elements distributed between server 110 and client device 108 and/or other computing devices in order to facilitate secure and/or effective communication between these computing devices.
[0070] As discussed above in connection with
[0071] As discussed above in connection with
[0072] In at least one embodiment, some or all of the system can be implemented as software written in any suitable computer programming language, whether in a standalone or client/server architecture. Alternatively, some or all of the system may be implemented and/or embedded in hardware.
[0073] Notably, multiple client devices 108 and/or multiple servers 110 may be networked together, and each may have a structure similar to those of client device 108 and server 110 that are illustrated in
[0074] In some embodiments, data within data store 106 may be distributed among multiple physical servers. Thus, data store 106 may represent one or more physical storage locations, which may communicate with each other via the communications network and/or one or more other networks (not shown). In addition, server 110 as depicted in
[0075] In one embodiment, some or all components of the system can be implemented in software written in any suitable computer programming language, whether in a standalone or client/server architecture. Alternatively, some or all components may be implemented and/or embedded in hardware.
[0076] Referring now to
[0077] Front-end 1301, which may be an interactive user interface presented on display screen 103 and accepting input via input device 102, may accept study parameters from user 100 and may display prompts, results, and other visual output for user 100.
[0078] Back-end 1302, which may run on processor 104, may receive study parameters collected by front-end 1301, and may initiate job 1303. Job 1303 may create study 1305 along with specified prompts 1304 to be presented to various LLM instances 1307 representing AI subjects. In at least one embodiment, LLM instances 1307 may be distinct instances of a single LLM. Responses received from LLM instances 1307 representing AI subjects may be stored in database 1309, which may be a postgres database stored on data store 106.
[0079] Job 1303 may fetch responses from database 1309 and may compute results 1306, which may include statistics from the extracted responses. The results may then be presented to user 100 via front-end 1301.
[0080] Referring now to
[0081] User interface 1401, which may be implemented by front-end 1301, may accept study setup parameters 1403 and/or other input 1402 from user 100. User interface 1401 may then display prompts, results, and other visual output for user 100, including results visualizations 1404.
[0082] Software code 1405, which may run on processor 104, may include study orchestration module 1407 for performing various functions such as statistics calculation 1408, AI subject (participant) generation 1409, API integration 1410, and prompt handling 1411. Statistics calculation 1408 module may generate results, which may then be displayed as results visualizations 1404 via user interface 1401. Software code 1405 may also include various utility functions 1406.
[0083] As described in more detail herein, software code 1405 may generate any number of AI subjects (simulated participants) 1412 for interaction with the study. AI subjects 1412 may be implemented, for example, using any number of LLM instances 1307. For example, in at least one embodiment, a separate LLM instance 1307 may implement each AI subject 1412. Each LLM instance 1307 may be a single instance of a particular LLM. In other embodiments, a single LLM instance 1307 may implement multiple AI subjects 1412.
Method
[0084] Referring now to
[0085] The method begins 300. In step 301, an AI study may be created or initialized. It may be entirely new, or it may be based on an existing study. In step 302, any number of AI subjects 1412 may be created, having characteristics that are helpful in analyzing the various stimuli. In at least one embodiment, AI subjects 1412 may represent simulated individuals; however, in other embodiments, they may represent simulated groups such as companies, families, organizations, governments, and/or other entities that may include multiple individuals.
[0086] In step 303, study setup and subject attributes may be translated into prompt grammar, using techniques described in more detail herein.
[0087] In step 304, AI subjects 1412 may then be assigned to control and treatment groups. In step 305, multimodal stimuli may be generated.
[0088] Next, in step 306, the system may match subjects to multimodal stimuli, including randomizing and counterbalancing. In step 307, for each stimulus, the system may then match subjects to one of multiple orders (counterbalanced or randomized) of measures.
[0089] In step 308, the system may translate stimuli, conditions (i.e., particular versions of stimuli that will be compared), and measures to the prompt grammar. In step 309, the system may create a prompt for a response schema that may be implemented in JavaScript Object Notation (JSON). Then, in step 310, the system may generate final prompts for AI subjects 1412.
[0090] In step 311, the system may concurrently send prompts to the large language models (LLMs) 1307 that implements AI subjects 1412, and store responses. As described in more detail below, matching metadata may also be stored, so that each response can be individually associated with its profile and other characteristics.
[0091] In step 312, responses may be converted to CSV format and/or to any other suitable format; for example, in at least one embodiment, responses may be stored in a database on data store 106. Finally, in step 313, statistical analysis may be replicated as desired using the stored information.
[0092] The method then ends 399.
[0093] Referring now to
[0094] The method begins 400. In step 401, a determination may be made as to whether existing study data will be used or a new study will be generated. If existing study data will be used, step 402 may be performed, in which input data that is already available may be ingested. Such input data can take any of various forms, including for example research study papers, product logs (interviews, CRM data, usage logs), and/or manual input from users.
[0095] If, in step 401, a new study will be generated, step 404 may be performed, in which the system may construct stimuli, AI subjects 1412, conditions, and/or the like, as described in more detail below.
[0096] As an example, in step 404, the system may create a study based on a real-time experiment using turn-based chat interaction, wherein low-latency experiments can be performed in the background with AI subjects 1412. A conversation may be facilitated to simulate responses to potential media messages, given a user's message and message history up to a particular point in time. The effect of this simulation is to allow quick experiments to be conducted during an interaction so that responses can be generated, based on the research that the system conducts, in what will seem to human interactants as real time.
[0097] In step 405, the system may create a StudyConfigurationObject, by translating research data and/or product logs. The StudyConfigurationObject may be an object representing parameters and characteristics of a study to be run. In at least one embodiment, the StudyConfigurationObject may allow for any or all of the following: [0098] An arbitrary number of simulated humans (also referred to as AI subjects, simulated subjects, virtual personas, subjects, or participants), simulated subject characteristics (e.g., age, gender, political affiliation, number of times going to a fine dining restaurant), and distributions (e.g., 50% male, 4 times a week, and/or the like). This may also include an arbitrary number of simulated subject behaviors (e.g., number of cigarettes smoked last week) and preferences (e.g., prefers hamburgers to salads). [0099] An arbitrary number of simulated and/or actual multimodal stimuli (e.g., text ads, Instagram posts, TikTok videos, product interfaces, and/or the like). [0100] An arbitrary number of measures, which may include questions of interest that each simulated subject should answer (e.g., How likely are you to buy this product?). [0101] An assignment of simulated subjects, stimuli, and measures to study conditions. For example, subjects with specific profiles (e.g., Age 29 Female in NYC who goes to fine dining three times a week and is expert in Microsoft Excel, grew up in Canada . . . ) may be assigned to specific stimuli with specific measures. [0102] A schema for outputting individual subject responses into a suitable format for analysis and/or storage in a database in a format that can be directly analyzed with stats libraries.
[0103] Referring now also to
[0104] In steps 406 and 407, the StudyConfigurationObject may be passed through a series of functions that translate documentation (such as research papers, product logs, and/or the like) into prompt grammar parts. An example of such a translation for a subject profile is You are a 22-year-old female. Your political affiliation is Democrat. You are an undergraduate student at a university in the USA. Now embody these characteristics as a subject in a research study. Deeply immerse yourself in the persona described.
[0105] An example of a translation for a stimulus is to encode an image of an Instagram post to base64 and insert it within grammar, as follows: Please look at the image and read the following text which is meant to accompany it as a caption. Caption: #ad New curated tracks by @DJSCL. Try new #Flow playlist for improved focus while at gym, driving, or coding!
[0106] An example of grammar chunks of measures pertaining to the above stimulus is as follows:
TABLE-US-00001 ActualUse: { questions: [ To what degree do you believe the influencer in the Instagram post actually uses the sponsored product in everyday life?, ], scale: 7-point Likert-type scale (1 = not at all; 7 = very much) }
[0107] Once all prompt grammar parts are constructed and the response schema is generated, in step 408, the prompts may be aggregated. Then, in step 409, each individual prompt may be sent to a single instance of a particular Large Language Model (LLM) 1307. In at least one embodiment, this may take place concurrently for all prompts. By sending each prompt to a different LLM instance 1307, the responses can be tailored to that specific AI subject 1412. For example, they can be uniquely personalized based on the AI subject's 1412 context, background, preferences, study condition, any specific measures, interaction history, and the like. In an alternative embodiment, however, a single LLM instance 1307 may implement multiple AI subjects 1412.
[0108] LLM instances 1307 implementing AI subjects 1412 may then process and respond to the prompts. In step 410, responses may be received from the LLM instances 1307. In step 411, the responses may be stored, for example in a database on data store 106, which may be located on device 101 or 108, or on a server 110. In at least one embodiment, as part of step 411, matching metadata may also be stored, so that each response can be individually associated with its profile and other characteristics.
[0109] Once the responses have all returned, in step 412, a function may be run that goes through all of the responses and dynamically creates a schema for the dataset. In step 413, a dataset may then be created (for example in a spreadsheet). In step 414, the dataset may be output for further statistical analysis. In at least one embodiment, desired statistics may be directly computed and the results may be output directly.
[0110] The method then ends 499.
Creation of AI Subjects
[0111] As described above in connection with step 302 of
[0112] The following is an example of pseudocode for the Participant class according to one embodiment:
TABLE-US-00002 Method .sub.init.sub.(self, characteristics) Iterates through characteristics, a dictionary of attributes. Dynamically sets each key-value pair in characteristics as an attribute of the Participant instance. Method create_prompt(self) Initializes a string representation with the participant's age, formatted as You are a {age} year old. Optionally appends additional participant information to prompt if available: Ethnicity, Gender, Country of Birth, Income, etc. Adds a sentence about the participant's preferred characteristics, e.g. knowledge of AI if available.
[0113] The following is an example of pseudocode for creating subjects (participants) according to one embodiment:
TABLE-US-00003 Initialize an empty list called participants. Calculate a scale factor by dividing number_participants by num_participants_in_paper. Loop number_participants times to create each participant: a. Create an empty dictionary characteristics for storing participant attributes. b. Directly assign an age to characteristics using the generate_age method. c. For each attribute and its distribution in participant_distributions: i. If the distribution is not empty, check if num_participants_in_paper is not zero and scale the distribution using scale_participant_distribution; otherwise, use the original distribution. ii. Generate the participant attribute using the possibly scaled distribution and assign it to the corresponding attribute in characteristics. d. If participant_random_characteristics is provided: i. For each characteristic and its options in participant_random_characteristics, assign a random choice of the options to the corresponding characteristic in characteristics. e. Create a Participant object with characteristics and append it to the participants list. Store the list of Participant objects in self.participants. Return the participants list.
[0114] Referring now to
[0115] The method begins 600. In step 601, an empty list of subjects (called participants) may be initialized. In step 602, a scale factor may be calculated by dividing number participants by num participants in paper. In step 603, a determination is made as to whether any more subjects are to be added (the loop will be executed number participants times). If so, steps 604 through 610 are performed; otherwise, the method proceeds to step 611.
[0116] In step 604, an empty repository called dictionary characteristics may be created for storing subject attributes. In step 605, age may be directly assigned to dictionary characteristics.
[0117] In step 606, attributes and distributions may be added. This may be performed by, for each attribute and its distribution in participant distributions, determining if the distribution is not empty. If it is not empty, a check may be made as to whether num participants in paper is not zero; the distribution may then be scaled using scale participant distribution. Otherwise, the original distribution may be used. The subject attribute may then be generated using the distribution (which may or may not be scaled), and may then be assigned to the corresponding attributed in characteristics.
[0118] In step 607, a determination is made as to whether random characteristics are provided. If so, step 608 is performed, in which, for each characteristic and its options in participant_random characteristics, a random choice of its options may be assigned to the corresponding characteristic.
[0119] In step 609, a participant object may be created using the determined characteristics, and in step 610, the new object may be appended to the participants list. The method then returns to step 603.
[0120] In step 611, the list of participant objects may be stored. In step 612, it may be returned.
[0121] The method then ends 699.
Creation and Assignment of Simulated Media Messages
[0122] As described above in connection with steps 305-307 of
[0123] The following is an example of pseudocode for creating and assigning simulated media messages according to one embodiment:
TABLE-US-00004 Iterate Over Participants: Loop through each participant in the list to individually assign stimuli based on their designated condition. Condition Assignment: Randomly assign a condition to each participant. This condition determines the specific stimuli (text or image) the participant will receive. Initialize Stimuli Collection: For each participant, prepare a list to collect their assigned stimuli. Initialize variables to track text and image stimuli. Process Conditions Stimuli Prepending and Appending: For each item of stimulus, prepend any required introductory text and append any concluding text as defined by the study's requirements. Finalize Participant Stimuli: Combine all assigned stimuli into a single, coherent presentation string for each participant.
[0124] Referring now to
[0125] The method begins 700. In step 701, a determination may be made as to whether there are any more AI subjects to which stimuli needs to be assigned. If so, steps 702 through 707 are performed; otherwise, the method ends 799.
[0126] For each AI subject, stimuli may be generated and individually assigned based on a designated condition(s) of the subject, as follows. In step 702, one or more condition(s) may be randomly assigned to the subject. Such condition(s) may determine the specific stimuli the subject will receive, which may include text, image(s), and/or any other type of stimuli.
[0127] In step 703, a collection of stimuli may be initialized for the subject. This may include, for example, a list for the subject to collect their assigned stimuli. In step 704, variables to track text and image stimuli may be initialized.
[0128] In step 705, condition(s) associated with the subject may be processed. In at least one embodiment, this may include the following substeps: [0129] Retrieve Condition Details: For each condition associated with the subject, relevant stimuli details (such as text or image URLs) may be extracted, for example based on a key associated with the condition. [0130] Text Stimuli Assignment: If the condition specifies text stimuli, appropriate text may be selected based on the condition's value. This substep may include, for example: [0131] Looking up a condition value within a predefined stimuli dictionary. [0132] Handling direct matches where the condition directly specifies the stimulus. [0133] Utilizing a fallback message if the specified condition or stimulus cannot be found. [0134] Image Stimuli Assignment: If applicable, a corresponding image URL for the condition may be retrieved.
[0135] In at least one embodiment, a structure may be established to allow image paths to be optionally associated with conditions and stored in a separate dictionary. This allows for flexible stimuli handling, since the structure may allow for easy inclusion of other types of stimuli (beyond text and image) by following a similar pattern of condition-based retrieval and assignment.
[0136] In step 706, for each item of stimulus, any required introductory text may be prepended, and any concluding text may be appended, as defined by the study's requirements. This step ensures that each stimulus is presented in a contextually appropriate manner.
[0137] In step 707, subject stimuli is finalized. In at least one embodiment, this may include combining all assigned stimuli into a single, coherent presentation string for each subject. This may include, for example, combining multiple text stimuli into a paragraph or list, and indicating the image to be viewed.
[0138] The described method provides improved extendibility, flexibility, and scalability. New types of stimuli (such as audio clips, videos, or interactive elements) can be integrated by extending the condition and stimuli dictionaries.
Creation of Measures
[0139] In at least one embodiment, in step 308, the system generates measures, which are the questions that are asked of the AI subjects, along with the desired scale for responses such as for example, semantic differential, 7-point Likert, and/or the like. Each measure can have any number of questions. The measures generated in step 308 may be based on input collected in steps 402 and/or 404.
[0140] The following is an example of pseudocode for creating measures from those defined in the study:
TABLE-US-00005 Iterate Over Measures: Loop through each measure defined in the study. Process Each Measure: Retrieve Questions and Scale: For each measure, extract the list of questions and the description of the scale used for responses. Randomize Question Order Personalize Questions: Conditional Prepending Track Original and Modified Questions Flexibility in Question Presentation
[0141] Referring now to
[0142] The method begins 800. Each measure defined in the study is processed in turn. In step 801, a determination may be made as to whether there are any more measures defined in the study that are to be processed. If so, steps 802 through 805 are performed; otherwise, the method ends 899.
[0143] In step 802, the next measure in the study may be selected for processing.
[0144] In step 803, the system may extract a list of questions and the description of the scale used for responses. The scale information contextualizes how responses should be interpreted (for example, by agreement, frequency, quality, and/or the like).
[0145] In step 804, the order of questions may be randomized. This may include shuffling the questions within each measure. This step helps to mitigate order bias, where the sequence of questions might influence responses.
[0146] In step 805, the questions may be personalized. This may include the following substeps: [0147] Conditional Prepending: For each question, if condition-specific prepend text exists, such as a short introduction or modification based on the subject's assigned condition, the system may replace a placeholder in the question with this text. This step allows for customization of questions based on the subject's condition, making the questions more relevant or specific to the subject's experience. [0148] Track Original and Modified Questions: In at least one embodiment, the system may maintain a mapping of original to modified questions. This may be useful for data analysis, allowing researchers to understand how each question was presented to subjects in the context of their condition. [0149] Flexibility in Question Presentation: In at least one embodiment, mechanisms may be provided for dynamically adjusting question text based on conditions assigned to subjects, thus providing improved adaptability of the survey to provide a tailored experience.
[0150] The use of a scale description alongside questions allows different sets of questions to utilize different response formats, offering flexibility in how subject attitudes, beliefs, or behaviors are assessed.
[0151] In at least one embodiment, the system can easily accommodate modifications to how questions are presented (for example, by adding new conditions that affect question wording) or the introduction of new scales for response collection. By maintaining a dynamic link between subject conditions and question presentation, the system is well suited to evolving research needs, including the introduction of new measures or the adaptation of existing ones to new study contexts.
Creation of Response Schema
[0152] In at least one embodiment, the system creates a response schema, which may be implemented in JavaScript Object Notation (JSON). The AI subjects may be configured to structure their responses in a JSON format to facilitate statistical analysis. One example of such a JSON format is as follows:
TABLE-US-00006 {yourResponses: [ {question: This restaurant has a strong record of environmentally friendly practices., yourAnswer: }, // 7-point Likert-type scale (1 = strongly disagree; 4 = neutral; 7 = strongly agree) {question: This restaurant supports good causes., yourAnswer: }, // 7-point Likert-type scale (1 = strongly disagree; 4 = neutral; 7 = strongly agree) {question: This restaurant behaves responsibly regarding the environment., yourAnswer: }, // 7-point Likert-type scale (1 = strongly disagree; 4 = neutral; 7 = strongly agree) {question: In resolving the problem, the restaurant gave me what I needed., yourAnswer: }, // 7-point Likert-type scale (1 = strongly disagree; 4 = neutral; 7 = strongly agree) ... ]}
[0153] By employing such an approach, the system may ensure that each question is encapsulated within an object that includes both the question text and a space for the participant's answer (yourAnswer: ). This design simplifies the process of parsing responses for statistical analysis, as each answer is directly associated with its corresponding question.
[0154] Any type of scale can be used, including for example: Likert scale, semantic differential, numerical, open-ended sentences, and/or the like.
[0155] By structuring responses in a JSON format, the system enforces consistency, which facilitates automated data collection and analysis. This consistency is useful for statistical software and scripts that process and analyze data, enabling efficient aggregation, summary statistics, and inferential statistics applications.
[0156] The JSON format also provides flexibility for expansion, by easily accommodating additional questions or modifications to existing ones. New questions can be added as objects within the yourResponses array without altering the overall structure, maintaining compatibility with analysis tools and scripts.
[0157] The structure may be optimized for statistical analysis, from basic descriptive statistics to more complex inferential analyses. Analysts can easily extract responses for each question across all participants, enabling comparisons, trend identification, and hypothesis testing regarding perceptions of the restaurant. In addition, the described format provides research flexibility, by supporting diverse research designs, including longitudinal studies where participant attitudes might be tracked over time, and/or comparative studies assessing the impact of interventions (such as, for example, changes in restaurant practices) on customer perceptions. Finally, the described format enables automated data processing; the JSON structure allows for seamless integration into data processing pipelines, enabling automated data cleaning, transformation, and analysis. This efficiency may be particularly beneficial in studies with large datasets, or those for which rapid analysis is necessary.
Generation of Final Prompts
[0158] As described in connection with step 310 of
[0159] Referring now to
[0160] The method begins 900. In step 901, the subject's stimulus prompt may be combined with a response schema, for example by appending the response schema to the subject's stimulus prompt, to construct a final study prompt. The response schema may define how responses should be structured, ensuring consistency across subjects.
[0161] In step 902, subject characteristics may be embedded as metadata. This step may include collecting characteristics of each participant, and dynamically assigning the subject's condition(s) to the metadata, enabling detailed tracking of the conditions' effects on responses.
[0162] In step 903, the prompt may be enriched with guidance. This step may include appending additional instructions or context (system_message_append) to the initial prompt (participant_prompt), creating a full system prompt. In this manner, AI subject 1412 may be guided as to how to embody the given characteristics during their response, enhancing the relevance and depth of their interaction.
[0163] In step 904, messages may be compiled with metadata. Each prompt, enriched with guidance and participant-specific characteristics, may be bundled with its corresponding metadata (including any relevant image URLs) into a structured format. This collection (messages with meta) facilitates easy access to both the prompts used and the detailed context of each interaction.
[0164] In step 905, a process may be executed to generate responses. The prepared prompts, now detailed with characteristics and enriched with guidance, may be fed into a process (such as an AI model) to generate responses. This step (self.run_gpt ( )) may include running the prompts through a generative model to obtain outputs that reflect the diverse characteristics and conditions of the subjects.
[0165] In step 906, responses from the subjects may be collected. As described above in connection with step 413 of
[0166] Referring now also to
[0167] The method then ends 999.
Statistical Analysis
[0168] In at least one embodiment, once responses have been collected, and a dataset 1000 has been generated and saved, statistical analysis may be performed using construct mappings to evaluate differences between conditions (such as, for example, media messages) based on responses received from the simulated subjects. These results can be integrated into a user-friendly decision-making user interface.
[0169] Referring now to
[0170] The method begins 1100. In step 1101, constructs may be defined (such as, for example, prejudiced motivation, purchase likelihood, and/or the like) using original survey question responses. In step 1102, a mapping may be constructed based on the defined constructs. In at least one embodiment, related questions may be grouped to form composite measures that reflect broader concepts or attitudes.
[0171] In step 1103, composite scores may be calculated. For each construct, composite scores for subjects may be determined by calculating the mean of responses to the associated questions. This process transforms individual item responses into aggregate scores that represent each construct for further analysis.
[0172] In step 1104, statistical testing may be performed. Depending on the research questions and data characteristics, appropriate statistical tests may be applied (such as, for example, t-tests, ANOVA, regression analysis) to compare construct scores across different conditions or groups. In this manner, an assessment may be made as to whether observed differences are statistically significant.
[0173] In step 1105, results may be integrated into a decision-making user interface (UI). Results of the statistical tests may be presented in a user-friendly interface that highlights key findings, such as to indicate condition was most effective or whether differences between groups were significant. In at least one embodiment, a decision-making feature may be incorporated that provides clear recommendations based on the test outcomes (for example, proceed with implementing a particular condition or strategy, halt an ineffective campaign, and/or the like). In at least one embodiment, for studies evaluating multiple stimuli, functionality may be included to rank them based on performance, according to the analysis results. This ranking aids in identifying the most effective stimuli for achieving the study's objectives.
[0174] The described framework supports not just t-tests but also a range of statistical methods tailored to different types of data and research hypotheses. Tests may be selected to align with the data distribution, the level of measurement, and the study design.
[0175] For categorical outcomes, chi-square tests or logistic regression may be appropriate. For continuous outcomes with more than two groups or conditions, ANOVA or multiple regression may be applied.
[0176] In at least one embodiment, an interactive dashboard may be provided, which may include any or all of the following features: [0177] Summary Statistics: Descriptive statistics for each construct by condition. [0178] Test Results Section: Clear presentation of statistical test outcomes, highlighting significant differences and offering interpretations in plain language. Data visualizations illustrate and summarize results. [0179] Decision Recommendations: A section that translates test results into actionable recommendations, using a color-coded system (e.g., green for go, red for no-go) to indicate the advisability of each condition. [0180] Stimuli Ranking: A ranked list of stimuli or conditions based on their effectiveness, as determined by the statistical analysis. This may be presented, for example, using bar charts and/or ordered lists. [0181] Tool Tips and Help Sections: Explanatory tool tips and help sections may be included that provide users with information on how to interpret the results and recommendations.
[0182] The method then ends 1199.
[0183] In at least one embodiment, based on the collected responses from the AI subjects, the system is able to automatically determine which of a number of different available techniques is best suited for generating the statistical report.
[0184] An example of raw output that may be generated by the described system is as follows. In at least one embodiment, such output may be presented in a simplified UI or may be immediately incorporated to surface a winning media message.
TABLE-US-00007 t = 17.996, df = 47.017, p-value < 2.2e16 alternative hypothesis: true difference in means between group Ad A and group Ad B is not equal to 0 95 percent confidence interval: 38.89841 31.07634 sample estimates: mean in group Ad A mean in group Ad B 51.94444 86.93182
[0185] In at least one embodiment, the system is able to automatically generate an additional stimulus for further research, based on the results of the statistical analysis. For example, if the study involves a comparison of advertisement A with advertisement B, the system is able to generate a statistical report showing which advertisement is more favorably received by various populations of AI subjects. In at least one embodiment, the system may be able to automatically generate an additional stimulus in the form for advertisement C, which may be designed to have even more favorable responses from the relevant population(s), based on the results obtained with respect to advertisements A and B.
User Interface
[0186] Referring now to
[0187]
[0188]
[0189]
[0190]
[0191]
[0192]
[0193]
[0194]
[0195]
[0196]
Use Cases
[0197] One skilled in the art will recognize that the described techniques can be applied to any of a number of different use cases. Each such use case can be applied using any of a number of different time domains for experiments, such as for example: [0198] Replication experiment, in which AI subjects are used to confirm and extend results in current literature. The techniques described herein may be used to provide more subjects, greater diversity in the sample, and/or a sampling of numerous media stimuli beyond examples in current literature. [0199] Original study, in which AI subjects may be used to complete a new study with a new research question. [0200] Contingency study, in which AI subjects may be used to perform an experiment that produces results that informs the next steps in an interaction as the interaction naturally unfolds. In at least one embodiment, the results may be generated in real-time or near real-time.
[0201] The following are examples of use cases for applicability of the techniques described herein.
AI Subjects in Mental Health Research
[0202] In an original smartphone intervention study, clinical researchers may wish to test two different strategies for mitigating the adverse effects of social media use on adolescent depression. One strategy monitors total screentime, regardless of the content being used, and another monitors smartphone use only within applications that enable social interactions. In at least one embodiment, the system is able to guide the choice of which strategy would be most effective in reducing depression symptoms. Specifics of the two strategies, which may be implemented in a smartphone app, may include the following: [0203] Screentime App: This intervention cues adolescents to spend less total time on their smartphones by tracking total screentime (regardless of the content of the screen) during one day and sending advisory text messages to users to reduce smartphone use when certain thresholds have been reached. [0204] Content App: This intervention monitors the categories of different applications being used on the smartphone in a single day (e.g., texting, social posting, shopping, browsing) and sends advisory text messages only when use thresholds for social applications are reached.
[0205] AI subjects may be sampled according to the protocol described herein, and sample smartphone interactions may be created (using AI) to be used in the experiment. Notably, in such a use case, the described system is able to sample a group of AI subjects, the real world counterpart for which may be extremely difficult to execute, because it is difficult to recruit adolescents for behavioral research. In addition, using conventional mechanisms, it would be difficult and time consuming to obtain complex approvals for use of human subjects; additional challenges exist due to data privacy and security requirements.
[0206] In at least one embodiment, the results of the AI experiment, using the prompt code for subject selection and statistical analytics, may guide selection of an appropriate clinical intervention.
[0207] In one example, the techniques can be used to test interactions for real-time mental health systems that may use avatars as chatbots during interactions. These interactions may engage a single individual (the patient) in an interaction about their mental health. For example, a chatbot may ask a question about mental health, the patient may respond, and the chatbot may then use that response to formulate the next comment in the interaction. In existing systems, however, the computer code that runs the interactions is generally guided by simple finite-state (if-then) models that do not allow responses to be customized to individual patients; alternatively, such systems may use general knowledge from the literature to create utterances that are targeted at average responses found to be useful in the literature.
[0208] The system described herein may provide improved results by anticipating the ability to do fast experiments, conducted within seconds during an interaction, so as to determine the next best utterances to use for the particular patient. This use may be specific to the individual, and the utterances may be developed via experiments conducted in real-time.
[0209] For example, a patient may tell the chatbot that they have experienced a traumatic event in the last week, such as the death of a family member. Rather than providing a canned response that is about family trauma, the described system may, in real-time, conduct an experiment to determine the best response for this particular individual, given all that is known up to that point about them, including, for example, demographics, personality, event circumstances, and/or the like. The system may sample appropriate possible responses, create different versions of what might be said, and determine via the prompt construction process an optimal response for this particular individual.
[0210] In such an application, the AI sample may be obtained in a number of different ways. For example, the system may sample other similar people for the experiment. Alternatively, if the interaction has already progressed over multiple turns, the sample may be constructed to predict future turns in the interaction for this single individual. Thus, either a cross-sectional sample (involving other similar people) or a longitudinal sample (involving other future time points for one individual) may be used; in yet another embodiment, the two approaches may be combined within in a single AI subject/stimulus design.
Product Design Use for AI Subjects/Stimuli
[0211] A typical product design process involves brainstorming multiple examples of product feature choices that are then sifted and winnowed via discussion or, if time allows, recourse to a research literature or single original study that a company designs to test alternatives.
[0212] In at least one embodiment, the techniques described herein may be used to enable a design process that conducts AI experiments about product alternatives.
[0213] For example, a product team may create several different options for a new brand identity for a home cleaning product. The new branding may include new options for a product name, logo, packaging, and/or advertising strategy. For each of the suggested options in all of the categories, AI experiments may be conducted using the techniques described herein, to allow the various options in each category to compete with one another. For example, different product name choices may be tested via AI with targeted consumers, different drawings of the product packaging options may be created and shown to AI subjects, and different sample advertisements for the product may be produced and tested. Using the techniques described herein, the total number of options may be tested, in near real-time, thus avoiding the need to make choices from intuitive discussions or from human subject experiments that may take weeks or months.
Generating AI Subjects for Replication and Extension of Social Scientific Findings
[0214] Guided by theory as well as practical developments, social scientists regularly examine how the content and structure of media-based stimuli may elicit different levels of cognitive, emotional, or behavioral responses. For example, researchers within the field of health communication may seek to investigate the relative effectiveness of different types of messaging for combatting vaping adoption in teens. Notably, many messaging campaigns have at times relied upon different types of messages, with some highlighting the negative aspects of engaging in risky behaviors, including effects on personal health, emotional distress resultingly experienced by loved ones, consequences for social connection, financial costs, or correlation with other undesirable outcomes (i.e., fear appeals, driving persuasion through psychological reactance). Alternatively, campaigns may create public service announcements or other materials in which the target behavior is promoted within a more positive context, highlighting benefits to health and social standing or prosocial outcomes. In these more positively toned materials, messages will often leverage humor to attract attention and enhance message comprehension, reduce counterarguing, and yield a more favorable attitude about the argument being made.
[0215] Often, the literature on the relative effectiveness of these different messaging strategies may yield mixed results, with effects potentially driven by contextual factors, like the particular subjects recruited, the specific behavior targeted, or the various attributes comprising a given stimulus. In at least one embodiment, the system described herein may be used to allow researchers to rapidly iterate and replicate existing studies, and/or generate and run new ones, helping to address concerns about the validity of previous findings. For example, researchers may sample AI subjects using techniques described herein, setting specific subject attributes (such as demographics, psychographics, and/or the like) and the relative distributions for each found within the study to be replicated. The system may process application instances of the specific stimuli used in the previous study and the specific response measures to be collected (such as Likert scale prompts, semantic differentials, open-ended questions, and/or the like). After confirming the exact experimental conditions to be compared and statistics to be computed, the replication study may then be run, using the techniques described herein, thereby obtaining in minutes new responses to different media message stimuli from custom AI subjects based on the original sample, and allowing a more meaningful point of comparison with the previous study.
[0216] If desired, the study may then be re-run with slightly altered characteristics of the AI subjects sample, thus avoiding the reliance of convenience sampling of non-representative subjects. Alternative measures may be used to capture target responses. In this capacity, the described system permits the researchers not only to potential replicate past findings but to inspect the generalizability of past findings to different population samples and examine the robustness of findings with respect to triangulated measures.
Generating AI Subjects and Stimuli for Original Research
[0217] When conducting studies in which subjects respond to stimuli, researchers must make several decisions, including what people to sample and who should see which stimuli. One important inquiry is to compare relative responses to different types of media messages, such as for example, fear appeals vs. humor appeals in public service announcements, gain frames vs. loss frames in the summary of a ballot resolution, and/or different types of disclaimer messages for debunking misinformation. Also important is the decision of which exact stimuli to show subjects. Unlike A/B testing scenarios in which a design team may be interested in the relative effectiveness of two preordained options, academic researchers often examine effects in light of theory-grounded concepts that are then rendered empirical through sampling of prospective stimuli.
[0218] In the case of responses to media stimuli, prospective messages may be selected from existing populations; these may represent many options that vary in terms of the causal variable of interest but also many other confounding attributes. Alternatively, researchers may specially construct custom messages, thereby allowing them to vary solely the causal variable of interest for the study. In either case, using conventional means, researchers may be limited with respect to access, should the ideal stimulus be behind a paywall or restricted to particular use given legal protections.
[0219] In at least one embodiment, the described system can be used to assist researchers by generating AI subjects for replication and extension of past work, and also by generating AI stimuli for wholly original research. For example, in the above-described example of examining the relative effectiveness of different anti-vaping messages, researchers who may be interested in conducting a novel study (rather than simply replicating or extending an existing study) may complete the user procedure described above, inputting the desired characteristics and distributions for AI subjects. However, instead of uploading extant stimuli for replication purposes, the researchers may input various parameters to consider for wholly novel stimuli to be generated. Such input may include descriptions of the relative depictions and emotional tone of the message contents, as well as any specific text elements. The described system may then generate a researcher-specified number of stimuli meant to capture the intended message type for each treatment group to be compared. Researchers may then manually select which resulting stimuli they would like to employ in the subsequent study run or, alternatively, have the system pre-test the different options with AI subjects in light of particular responses variables (with respect to presence of the intended target variable and absence of potential confounds).
[0220] The described approach to selecting AI generated stimuli may provide significant improvements over conventional pilot testing which, using conventional research practices, is typically costly in terms of time and effort. Regardless of stimulus selection approach, as with the previous example, the researchers may then confirm the statistics to be computed and run the study. In this manner, the researchers may be able to complete stimulus sampling and pre-testing as well as rapid data collection and analysis for an original study.
Advantages
[0221] The above-described techniques provide several advantages over known methods for testing human responses to media stimuli. For example: [0222] The described system and method provide the ability to create a prompt grammar for any arbitrary subject characteristics given documentation such as a research paper or product log, and/or for any arbitrary media stimuli characteristics (whether replicated or wholly original), given such documentation. [0223] The described system and method provide the ability to create a prompt grammar for any arbitrary subject measures given documentation such as a research paper or product log. [0224] The described system and method provide the ability to assign subjects to interact with specific or random stimuli and/or measures. In at least one embodiment, experiments may be performed very quickly, and using a specified interactive sequence. [0225] The described system and method provide the ability to generate a quantitative dataset with the subjects' responses to the measures. [0226] The described system and method provide the ability to generate a recommendation for which stimulus to be used, and/or to generate a new stimulus to be used.
[0227] The present system and method have been described in particular detail with respect to possible embodiments. Those of skill in the art will appreciate that the system and method may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms and/or features may have different names, formats, or protocols. Further, the system may be implemented via a combination of hardware and software, or entirely in hardware elements, or entirely in software elements. In addition, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
[0228] Reference in the specification to one embodiment or to an embodiment means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. The appearances of the phrases in one embodiment or in at least one embodiment in various places in the specification are not necessarily all referring to the same embodiment.
[0229] Various embodiments may include any number of systems and/or methods for performing the above-described techniques, either singly or in any combination. Another embodiment includes a computer program product comprising a non-transitory computer-readable storage medium and computer program code, encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-described techniques.
[0230] Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a memory of a computing device. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It may be convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it may also be convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
[0231] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it may be appreciated that throughout the description, discussions utilizing terms such as processing or computing or calculating or displaying or determining or the like, refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
[0232] Certain aspects include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions may be embodied in software, firmware and/or hardware, and when embodied in software, may be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
[0233] The present document also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computing device. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, DVD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMS, EEPROMs, flash memory, solid state drives, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Further, the computing devices referred to herein may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
[0234] The algorithms and displays presented herein are not inherently related to any particular computing device, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. In addition, the system and method are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings described herein, and any references above to specific languages are provided for disclosure of enablement and best mode.
[0235] Accordingly, various embodiments include software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device may include, for example, a processor, an input device (such as a keyboard, mouse, touchpad, track pad, joystick, trackball, microphone, and/or any combination thereof), an output device (such as a screen, speaker, and/or the like), memory, long-term storage (such as magnetic storage, optical storage, and/or the like), and/or network connectivity, according to techniques that are well known in the art. Such an electronic device may be portable or non-portable. Examples of electronic devices that may be used for implementing the described system and method include: a mobile phone, personal digital assistant, smartphone, kiosk, server computer, enterprise computing device, desktop computer, laptop computer, tablet computer, consumer electronic device, or the like. An electronic device may use any operating system such as, for example and without limitation: Linux; Microsoft Windows, available from Microsoft Corporation of Redmond, Washington; MacOS, available from Apple Inc. of Cupertino, California; iOS, available from Apple Inc. of Cupertino, California; Android, available from Google, Inc. of Mountain View, California; and/or any other operating system that may be adapted for use on the device.
[0236] While a limited number of embodiments have been described herein, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised. In addition, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the subject matter. Accordingly, the disclosure is intended to be illustrative, but not limiting, of scope.