Quality of experience monitoring system and method for information content distributed through a telecommunications network

Abstract

A method for estimating a quality of experience of a user receiving an information content stream distributed through a telecommunications network is provided comprising: generating a user behavior model, depicting a behavior of the user in the interaction with the telecommunications network; implementing the user behavior model by selecting combinations of user behavior; providing monitoring units adapted to perform measures on specific parameters of the information content stream; setting the monitoring; assigning to the monitoring units test cycles comprising interaction operations with the telecommunications network; generating for each monitoring unit a corresponding script to be executed by the monitoring unit; at the monitoring units, carrying out the test cycles by running the generated scripts, and carrying out measurements on parameters of the information content stream received in response to the test cycles to obtain acquired measured values, and calculating quality of experience indicators by processing the acquired measured values.

Claims

1. A method for estimating a quality of experience of a user receiving an information content stream distributed through a telecommunications network, the method comprising: generating a user behavior model, depicting a behavior of a user in an interaction with the telecommunications network, by considering a corresponding set of user behavior dimensions, each user behavior dimension being a descriptor describing a corresponding feature of the behavior; implementing the user behavior model by selecting combinations of user behavior dimensions of the set of user behavior dimensions; providing monitors to perform measures on specific parameters of the information content stream and to emulate the behavior of the user in the interaction with the telecommunications; setting the monitors, the setting the monitors comprising setting a number, type and/or location of the monitors based on the selected combinations of user behavior dimensions; assigning to the monitors test cycles comprising interaction operations with the telecommunications network to be carried out by the monitors; generating for each monitor a corresponding script to be executed by the monitor based on the selected combinations of dimensions and the assigned test cycles, the script comprising indications about the interaction operations of the assigned test cycles; at the monitors, carrying out the test cycles by running the generated scripts, and carrying out measurements on parameters of an information content stream received in response to the test cycles to obtain acquired measured values, and calculating quality of experience indicators by processing the acquired measured values.

2. The method of claim 1, further comprising: providing a set of different test typologies, and partitioning a test time period into a set of time slots, wherein: the assigning to the monitors test cycles comprises generating a second table comprising a plurality of second elements, each second element providing an indication of a number of repetitions of test cycles of a corresponding test typology to be carried out during a corresponding time slot.

3. The method of claim 2, further comprising generating a third table listing, for each monitor, at least one among: information about a name of the monitor; information about a geographical position of the monitor; information about an interconnection configuration of the monitor; information about software and/or hardware equipment of the monitor; details about the interaction operations to be carried out and how the interaction operations are arranged in test cycles; a test calendar comprising information specifying which test cycle(s) to be carried out by monitor and when, wherein the generating for each monitor a corresponding script comprises generating the script based on the third table.

4. The method of claim 3, wherein the generating the third table comprises generating the third table based on the first elements of the first table and on the second elements of the second table.

5. The method of claim 2, wherein the implementing the user behavior model further comprises: for each test cycle to be carried out during a time slot, setting a corresponding maximum duration of the test cycle; and setting the second elements of the second table based on the corresponding maximum durations and on a duration of the time slots.

6. The method of claim 5, further comprising verifying correct operations of the monitors by comparing for each monitor a current time with the maximum duration of the test cycle carried out by the monitor.

7. The method of claim 2, further comprising generating a fourth table listing: a last data transmission carried out by a monitor; a new data transmission awaited by a monitor; a last configuration downloaded by a monitor; a last task wait signal by an agent, wherein: the method further comprises verifying a coherence of the acquired measured values based on the fourth table.

8. The method of claim 1, wherein each user behavior dimension of the set of user behavior dimensions may assume a plurality of different user behavior dimension values, the implementing the user behavior comprising generating at least one first table comprising a plurality of first elements, each first element quantifying a relationship among dimension values of the dimensions of the set of user behavior dimensions.

9. The method of claim 8, wherein the setting the number, type and/or location of the monitors comprises setting the number, type and/or location of the monitors according to elements of the first table.

10. A system for estimating a quality of experience of a user receiving an information content stream distributed through a telecommunications network, the system comprising a plurality of monitors to perform measures on specific parameters of the information content stream, and a central circuitry interfaced with the monitors, the central circuitry being configured to: generate a user behavior model, depicting a behavior of a user in an interaction with the telecommunications network, by considering a corresponding set of user behavior dimensions, each user behavior dimension being a descriptor describing a corresponding feature of the behavior; implement the user behavior model by selecting combinations of user behavior dimensions of the set of user behavior dimensions; assign to the monitors test cycles comprising interaction operations with the telecommunications network to be carried out by the monitors; generate for each monitor a corresponding script to be executed by the monitor based on the selected combinations of dimensions and based on the assigned test cycles, the script comprising indications about the interaction operations of the assigned test cycles, wherein the monitors are configured to carry out the test cycles by running the generated scripts, and carry out measurements on parameters of the information content stream received in response to the test cycles to obtain acquired measured values, and the central circuitry is further configured to calculate quality of experience indicators by processing the acquired measured values.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) These and other features and advantages of the present invention will better appear from the following detailed description of some embodiments thereof, provided merely by way of non-limitative examples. The description should be read in conjunction with the attached drawings, wherein:

(2) FIG. 1 schematically shows a structure of a system according to an embodiment of the present invention, for obtaining estimations of the quality of experience perceived by users enjoying multimedia contents distributed through a telecommunications network;

(3) FIG. 2 is a flow chart depicting the main phases of a procedure carried out by the system of FIG. 1 for monitoring the multimedia content stream and estimating the quality of experience according to an embodiment of the present invention.

(4) Making reference to the drawings, in FIG. 1 an exemplary scenario 100 is schematically shown wherein the present invention is exploitable. The scenario 100 considered is one in which audio and/or video contents, e.g. multimedia contents like for example music, clips, movies, VoIP phone calls, video calls, IPTV, VOD, shared contents and the like are distributed through a telecommunications network 110, e.g. the Internet; hereinafter, without prejudice to the generality of the invention, the distributed contents will be referred to as multimedia contents or information content. The multimedia contents are made available and distributed by one or more content providers 105 in the form of data streams (e.g., streams of data packets); the generic multimedia content stream, denoted in the figure as 115, may include one or more content components (e.g., one or more audio components, and/or one or more video components), distributed by a same or by different content providers. The multimedia content stream 115 is transported by the telecommunications network 110 to a user equipment 120, connected to the network 110; exploiting the user equipment 120, the user can enjoy the multimedia content. The user equipment 120 may be a personal computer (as in the example depicted), a notebook, a laptop, a tablet, a cellular phone, a smart phone, a VoIP phone, a set-top box for IPTV, associated to a TV set, a portable media player, and in general any device having data processing capabilities and capable of communicating over the telecommunications network 110.

(5) FIG. 1 also shows schematically the structure of a system 125 according to an embodiment of the present invention, for monitoring the multimedia content stream 115 and estimating a Quality of Experience (QoE) as perceived by the user enjoying the multimedia content using the user equipment 120.

(6) The system, globally denoted 125, includes one or more monitoring units, referred to as agent units 130(1), 130(2), . . . , 130(i), . . . executing specifically designed software adapted to perform measures on specific parameters of one or more of the components of the multimedia content stream 115.

(7) The agent units 130(i) are geographically distributed in various positions of the area wherein the telecommunications network 110 is located, in such a way to emulate user equipment 120 connected to the network 110. Each agent unit 130(i) is provided with hardware capable of emulating the user experience related to a specific use of a service provided by a content provider 105. For example, the agent units 130(i) may be personal computers, notebooks, laptops, tablets, smart phones, a set-top boxes for IPTV, portable media players, and so on. The agent units 130(i) are configured to run the same software used by the end users of the user equipment 120, as well as a specific software for carrying out tests emulating the behavior of end users exploiting said user equipment 120 for interacting with the telecommunications network. Each agent unit 130(i) is also configured to run a further specific software adapted to carry out measures on parameters (e.g., bandwidth, jitter, delay, response time) of the multimedia content stream 115 received during the execution of the tests. In order to protect the results of the measurements carried out by the agent units 130(i) from possible alterations, the agent units 130(i) are advantageously hardened.

(8) The system 125 comprises a central unit 140 which is interfaced with the agent units 130(i) by means of a dedicated monitoring network 145 for acquiring and processing the measurements collected by the agent units 130(i). For this purpose, the central unit 140 is provided with an acquisition unit 150 configured to acquire values of the parameters of the multimedia content stream 115 measured by the agent units 130(i) during the execution of the tests (hereinafter, simply referred to as measured values); the acquisition unit 150 is coupled with a processing unit 152 configured to process the measured values acquired by the acquisition unit 150 for calculating corresponding QoE indicators indicative of the QoE as perceived by the user enjoying the multimedia content. Alternatively, a single unit may be provided carrying out the function of both the acquisition unit 150 and the processing unit 152. The central unit 140 further comprises a test scheduler unit 153 adapted to configure the tests to be carried out by the agent units 130(i) and an agent configuration unit 154 for configuring the agent units 130(i) in such a way to run the configured tests in order to emulate the behavior of end users.

(9) The central unit 140 is further provided with a measured values repository unit 155, e.g., a memory unit, configured to store the acquired measured values, preferably only after the validity of the acquired measured values has been verified, and to store information related to the operations carried out by the corresponding agent units 130(i).

(10) The central unit 140 further comprises a test and configuration repository unit 160, e.g., a memory unit, configured to store the details of the tests to be carried out by the various agent units 130(i) and their configurations.

(11) The system 125 comprises a managing unit 165 interfaced with the central unit 140 and the agent units 130(i) through the monitoring network 145 for the management of the acquisition unit 150, the measured values repository unit 155, the test repository unit 160 and the agent units 130(i).

(12) One or more measure consulting units 170 (only one illustrated in figure) are interfaced with the central unit 140 for allowing users of the system 125 to consult the acquired measured values and the corresponding QoE indicators calculated by the central unit 140.

(13) FIG. 2 is a diagram illustrating the main phases of a procedure 200 carried out by the system 125 for monitoring the multimedia content stream 115 and estimating the QoE according to an embodiment of the present invention.

(14) The first phase 210 of the procedure 200 provides for defining a user behavior model of the users to be exploited for configuring the tests to be carried out by the agent units 130(i), and then for allocating the agent units 130(i) based on such user behavior model. The Applicant has observed that efficient user behavior models need to take into account in a very detailed way all the steps that customers typically have to carry out when they are using desired services provided by the content providers 105.

(15) According to an embodiment of the present invention, the user behavior model depicting a user behavior to be emulated is defined by considering a corresponding set of user behavior dimensions of the latter (hereinafter, simply dimensions), wherein each dimension is a descriptor describing a corresponding feature of the user behavior to be emulated. Each dimension may assume a corresponding plurality of different values, referred to as dimension values.

(16) According to an embodiment of the present invention, four classes of dimensions are provided: Interaction type dimension class: the dimensions of such class relate to the definition of which type of phenomenon is measured, such as for example: an application, a service class, an http page, a download, a content usage. Temporal dimension class: the dimensions of such class relate to the time during which the phenomenon to be measured is carried out, such as for example within a day, and/or within a week and/or within a month. Device and network dimension class: the dimensions of such class relate to the type of hardware (personal computer, smart phone, tablet . . . ), operating system and programs used by users for accessing the contents. Geographical dimension class: the dimensions of such class specify how and where the users are geographically distributed, as well as the type of access employed by the latters (ADSL, wireless, mobile, and so on).

(17) The concepts of the present invention apply also if a different number of dimension classes is considered, such as for example with a lower number of classes or with additional classes.

(18) This peculiar way of defining the user behavior model through a subdivision of the user behavior in terms of dimensions allows to define the typology of the tests to be carried out, starting from a series of indicators about the QoE perceived by a user for a specific service.

(19) According to an embodiment of the present invention, for each interaction type, the single steps with which such interaction type may be implemented are defined in terms of scripts to be automatically carried out by the agent units 130(i) for correctly assigning the task thereto based on a respective calendar (this aspect will be described in greater detail in the following of the description).

(20) According to an embodiment of the present invention, the first phase 210 of the procedure 200 provides for generating a user behavior model by selecting specific dimensions within the abovementioned classes of dimensions to describe corresponding features of interest of the user behavior to be emulated. Then, the dimensions values for these selected dimensions are selected, for example based on log analysis, survey results, market data, focus groups, user interviews, and so on. From these data it is possible to obtain a minimum emulation scenario, which will be enriched by considering the objectives of the specific measure campaign.

(21) An example of a user behavior defined in terms of dimensions according to the present invention is disclosed below.

(22) User behavior of a group of users residing in Torino, Milano, Firenze, Roma which make accesses by means of a personal computer of type x, y, z with operating system w1, w2 through an ADSL line obtained through the commercial offer a, b, c every working day at a portal p for re-watching from 20:00 to 21:00 the afternoon program xxx and for watching the live program broadcasted by the TV channel u from 21:00 to 24:00.

(23) Possible dimensions of said exemplificative user behavior are the following ones: Dimension City (belonging to the Geographical dimension class), having the following dimension values: Torino, Milano, Firenze, Roma; Dimension Device (belonging to the Device and network dimension class), having the following dimension values: personal computer of type x, personal computer of type y, personal computer of type z; Dimension Operating system (belonging to the Device and network dimension class), having the following dimension values: operating system w1, operating system w2; Dimension Connection (belonging to the Geographical dimension class), having the following dimension values: ADSL line obtained through the commercial offer a, ADSL line obtained through the commercial offer b, ADSL line obtained through the commercial offer c; Dimension Day (belonging to the Temporal dimension class), having the following dimension value: every working day; Dimension Access (belonging to the Interaction type dimension class), having the following dimension value: portal p; Dimension Time1 (belonging to the Temporal dimension class), having the following dimension value: from 20:00 to 21:00; Dimension Interaction1 (belonging to the Interaction type dimension class), having the following dimension value: program xxt; Dimension Interaction2 (belonging to the Interaction type dimension class), having the following dimension value: live program broadcasted by the TV channel u; Dimension Time2 (belonging to the Temporal dimension class), having the following dimension value: from 21:00 to 24:00.

(24) As can be seen from this example, according to an embodiment of the present invention, a user behavior model may be defined by selecting more (different) dimensions belonging to a same class (e.g., the dimensions City and Connection belong to the Geographical dimension class).

(25) According to an embodiment of the present invention, user behavior models can be also defined in case the whole context knowledge is not known (for example because it is not possible to have all the real data of the desired user behavior model), or for emulating new application scenarios (for example because a new service, device, or network technology is launched). In these cases, some of the dimensions are lacking, and are replaced with corresponding variables.

(26) The second phase 220 of the procedure 200 provides for implementing the user behavior model generated at phase 210 by defining which combinations of dimensions are to be used among the ones selected in phase 210 in order to form corresponding dimension tables. Such operations are carried out by the test scheduler unit 153.

(27) According to an embodiment of the present invention, a dimension table DT is a table having a matrix dimension of n, which expresses the relationships among the possible combinations of the dimension values of n dimensions chosen among the dimensions selected in phase 210. The definition of which combinations may be used depends on the predominant features of the desired measure. The way the dimension are selected to be combined is carried out based on the objective of the campaign to be implemented.

(28) Making reference to the example at issue, four dimensions may be for example combined to generate two bi-dimensional dimension tables DT: a first dimension table DT1 combining the dimension Access with the dimension Day, and a second dimension table DT2 combining the dimension City with the dimension Device. According to another example, the same four dimensions may be combined differently for obtaining two different bi-dimensional dimension tables DT: a first dimension table DT1 combining the dimension City with the dimension Day, and a second dimension table DT2 combining the dimension Access with the dimension Device. According to a still further example, the same four dimensions may be combined to form a single four-dimensional dimension table DT.

(29) Once the dimensions of a dimension table DT have been selected, the dimension table DT is generated. Specifically, according to an embodiment of the present invention, each element of a n-dimensional dimension table DT corresponding corresponds to a specific n-tuple of dimension values from the user behavior model. The elements of dimension tables DT are set using different instruments, such as log analysis, surveys, and/or a priori definitions of data relating to the hypothetical scenario to be modeled. According to an embodiment of the present invention, in case more than one dimension tables DT are formed, one of them is elected to be a main dimension table MDT. The main dimension table MDT is the dimension table DT that corresponds to the most important combinations of dimensions for the purposes of the desired monitoring.

(30) Making reference to the example at issue, a possible main dimension table DTM may be a bi-dimensional dimension table DT corresponding to the combination of the dimension City (dimension values: Torino, Milano, Firenze, Roma) with the dimension Device (dimension values: personal computer of type x, personal computer of type y, personal computer of type z). An example of such main dimension table DTM corresponding to a very simplified scenario is depicted hereinbelow:

(31) TABLE-US-00001 City: Torino City: Milano City: Firenze City: Roma Device: pc x 1 2 1 1 Device: pc y 1 1 1 1 Device: pc z 0 4 0 2
wherein the element 1 corresponding to the pair of dimension values (Torino, pc x) means that in the considered scenario, in the City of Torino resides one user possessing a personal computer of type x, the element 2 corresponding to the pair of dimension values (Milano, pc x) means that in the considered scenario, in the City of Milano reside two users possessing a personal computer of type x, and so on.

(32) According to an embodiment of the present invention, the number, type and/or location of the agent units 130(i) of the system 125 are set based on the elements of the main dimension table DTM, in such a way that the measures carried out by the agent units 130(i) are performed in a way that suitably fits the user behavior to be emulated.

(33) For example, making reference to the example at issue, according to an embodiment of the present invention, for each column of the main dimension table DTM, i.e., for each specific dimension value of the dimension City, a number na of agent units 130(i) equal to the sum of elements of such column is located at the location corresponding to said dimension value. According to this embodiment of the invention, in the example at issue there will be na=2 agent units 130(i) in Torino, na=7 agent units 130(i) in Milano, na=2 agent units 130(i) in Firenze, and na=4 agent units 130(i) in Roma.

(34) According to another embodiment of the invention, for each column of the main dimension table DTM, i.e., for each specific dimension value of the dimension City, a number na of agent units 130(i) proportional to the sum of elements of such column is located at the location corresponding to said dimension value. Using for example a same proportionality coefficient of 2 for every column of the main dimension table DTM, in the example at issue there will be na=4 agent units 130(i) in Torino, na=14 agent units 130(i) in Milano, na=4 agent units 130(i) in Firenze, and na=8 agent units 130(i) in Roma.

(35) According to a still further embodiment of the invention, a different proportionality coefficient may be used for each (group of) column(s) of the main dimension table DTM. For example, if the monitoring of the cities of Torino, Milano and Roma is considered more important than the one of the city of Firenze, the proportionality coefficients for the former three may be advantageously set higher than the one for the latter one.

(36) Making again reference to the example at issue, once the number and location of the agent units 130(i) are set, the type of agent units 130(i) for each city is determined based on how the elements of the corresponding column are distributed in the rows of the main dimension table DTM. For example, the na agent units 130(i) located in the city of Torino may be subdivided in the following way: a number na/2 of agent units 130(i) will be of the type personal computer of type x; a number na/2 of agent units 130(i) will be of the type personal computer of type y; a number 0 of agent units 130(i) will be of the type personal computer of type z.

(37) More generally, according to an embodiment of the present invention, the test scheduler unit 153 sets the number, type and/or location of the agent units 130(i) of the system 125 using the way the elements of the main dimension table DTM are distributed as guidelines.

(38) Once the number, type and/or location of the agent units 130(i) are set, the test scheduler unit 153 is configured to assign to each agent unit 130(i) specific tests to be carried out for emulating the user behavior. Said tests are structured so as to comprise test cycles each one comprising in turn interaction operations with the telecommunications network. As mentioned in the foregoing, the same agent units 130(i) are further configured to carry out measures on parameters of the multimedia content stream 115 received during said tests.

(39) The test scheduler unit 153 assigns to each agent unit 130(i) the typologies of the tests to be carried out, and the number of test repetitions to be carried out during corresponding time slots of a test time period. For example, the test time period may correspond to a day.

(40) For this purpose, according to an embodiment of the present invention, a further tablereferred to as test table TTis generated, having a column for each time slot of the day, and a row for each test typology. Each element (referred to as interaction frequency) of the test table TT corresponding to a specific (time slot, test typology) pair indicates the number of repetitions of said specific test typology during said specific time slot of the day.

(41) An exemplary test table TT corresponding to a case in which the day is subdivided in k=4 time slots TS(k) (TS(1)=Morning, TS(2)=Afternoon, TS(3)=Evening, TS(4)=Night) and in which j=3 different test typologies Ttype(z) (z=1, 2, 3) are provided is illustrated herein below:

(42) TABLE-US-00002 TS(1) TS(2) TS(3) TS(4) Ttype(1) x(1, 1) x(1, 2) x(1, 3) x(1, 4) Ttype(2) x(2, 1) x(2, 2) x(2, 3) x(2, 4) Ttype(3) x(3, 1) x(3, 2) x(3, 3) x(3, 4)
wherein x(1,1) is the number of repetitions of a test cycle TC(1) of the test typology Ttype(1) to be carried out during the Morning time slot TS(1), x(1,2) is the number of repetitions of a test cycle TC(1) of the test typology Ttype(1) to be carried out during the Afternoon time slot TS(2), and so on.

(43) A test cycle TC(z) generally provides for carrying out interaction operations sets IOS(n) each one comprising a sequence of elementary test operations emulating the behavior of a user when interacting with the network 110 according to a specific behavior model. Non limitative examples of interaction operation sets IOS(n) may comprise the operations required for the transmission of a file from a server (e.g., connecting to the server through ftp, exchanging credentials, logging in, setting the correct directory, defining the way such file transmission has to be carried out, actually transmitting the file, logging out), operations required for using the browser for reading a first email and sending a second email (e.g., opening a browser on a provider page, opening login page, logging in, opening the inbox directory, reading the first email, selecting a new email option, selecting the recipients, writing the second email text, sending the second email, logging out), and so on.

(44) Each one of said interaction operation sets IOS(n) comprised in a test cycle TC(z) may be repeated within the test cycle TC(z) with a corresponding interaction frequency If(n). Therefore, the duration of a test cycle TC(z) is obtained by summing the durations of the interaction operation sets IOS(n) comprised in the test cycle TC(z), each one multiplied by the corresponding interaction frequency If(n).

(45) For each test typology Ttype(z), a minimum mT(z) and a maximum MT(z) duration of the corresponding test cycle TC(z) are set. In some cases, such as in case the multimedia content is a movie or a song, such two durations are equal.

(46) The number of repetitions x(z,k) of a test cycle TC(z) of the test typology Ttype(z) to be carried out during a time slot TS(k) is set taking into consideration the ratio between the duration of such time slot TS(k) and the maximum duration MT(z) of the test cycle TC(z).

(47) Once the number of repetitions x(z,k) of test cycles TC(z) that can be carried out in each time slot TS(k) is set, the available agent units 130(i) are accordingly allocated.

(48) The third phase 230 of the procedure 200 is a configuration phase carried out by the agent configuration unit 154 for translating the user behavior models previously defined in computer program instructions (scripts) that can be executed by the agent units 130(i), and for making them available (or for directly providing them) to the agent units 130(i).

(49) The agent configuration unit 154 is adapted to configure the agent units 130(i) by generating scripts which the agent units 130(i) are capable of automatically running starting from a predefined starting time and for a predefined duration.

(50) In order to correctly operate, the agent units 130(i) have to be of the correct hardware typology, possess the correct software and interconnection capability, and be physically located in the selected location. These checks are partially provided in the configuration file of the central unit 140 and partially carried out by a human actor.

(51) The first operation carried out by the configuration unit 154 according to an embodiment of the present invention is to collect required information from the dimension tables DT and the test table TT generated by the test scheduler unit 153, arranging them in a corresponding configuration table CT such as the following one.

(52) TABLE-US-00003 C(1) C(2) C(3) 130(1) c(1, 1) c(1, 2) x(1, 3) 130(2) c(2, 1) c(2, 2) x(2, 3) . . . . . . . . . . . . 130(i) c(i, 1) c(i, 2) x(i, 3)

(53) According to this embodiment of the invention, the configuration table CT comprises a row for each agent unit 130(i). For each agent unit 130(i) (i.e., for each row), the configuration table CT provides for arranging information corresponding thereto in three respective classes, each one corresponding to a column of the configuration table CT: agent information class (column C(1)), test information class (column C(2)), and test calendar information class (column C(3)).

(54) The agent information class comprises information c(i, 1) about the agent units 130(i), such as their names, the geographical location thereof, the interconnection configuration thereof, details on their hardware and software equipment, and IP address of the monitoring network 145.

(55) The test information comprises information c(i, 2) relating to specific details about the interaction operation sets IOS(n), and how they are arranged in test cycles TC(z) to be carried out by the agent units 130(i).

(56) The test calendar information class comprises information c(i, 3) relating to calendars of the tasks to be carried out during the tests, which specifies which agent unit(s) 130(i) has(have) to execute which test cycle TC(z) and when.

(57) The concepts of the present invention also apply to the case in which a different set of information are used to generate the configuration table CT, such as a subset of the information listed above.

(58) Making for example reference to a test cycle TC(z) comprising an interaction operation set IOS(n) for the transmission of a file from a server through ftp to be carried out by an agent unit 130(i), the row corresponding to such agent unit 130(i) may comprise the following information: c(i, 1):

(59) Name=agente A

(60) NameDescription=tlc00312060002 c(i, 2):

(61) 0_connessione=ADSL

(62) 0_RASClassID=101

(63) 0_tipoPOP=tlc00301

(64) 0_localita=Torino

(65) 0_numero=011xxx

(66) 0_username-yyy

(67) 0_password=www

(68) 0_FTPClassID=1

(69) 0_FTPNome=193.104.137.129

(70) 0_FTPusername=jjj

(71) 0_FTPpassword=kkk

(72) 0_FTPtotalTMO=300

(73) 0_FTPtestmode=GET

(74) 0-FTPremotefile=download/pippo c(i, 3):

(75) pause=20

(76) group={ }

(77) startat=04/12/2010 00:15

(78) endat=31/12/2010 23:45

(79) According to an embodiment of the present invention, based on the information arranged in the configuration table CT, the configuration unit 154 automatically generates a corresponding script for each agent unit 130(i), which comprises indications about the specific interaction operation set IOS(n) to be executed by the agent units 130(i) during the test cycle TC(z).

(80) Then, the configuration unit 154 stores such calendars in a memory area whose address and access credentials are known to the agent units 130(i). For example, according to an embodiment of the present invention, said memory area may be located in a storage area 190 located on the central unit 140 itself (see FIG. 1).

(81) Generally, the calendars may be different for each agent unit 130(i). According to an embodiment of the present invention, the calendars of the agent units 130(i) may be changed at any time.

(82) An example of a possible script for an agent unit 130(i) may be structured in such a way to comprises the following four sections: an identity section, which comprises a name and an identification code of the agent unit 130(i); a credential section, which comprises credentials of the servers for the retrieval of the calendars and the results storing; a task section, which comprises instructions relating to the tasks to be carried out by the agent unit 130(i); a calendar section, which comprises calendars of the tasks to be carried out during the tests.

(83) Each agent unit 130(i) is activated by copying, e.g., manually, an initial configuration comprising the address and the access credentials to the storage area 190 wherein the calendars are stored. Then, each agent unit 130(i) periodically accesses to the storage area 190 through the monitoring network 145, using the known credentials. Each agent unit 130(i) verifies if new tasks have been assigned thereto and, in the positive case, it retrieves them and, after the operations are concluded, it provides a corresponding advise to the system.

(84) The fourth phase 240 of the procedure 200 according to an embodiment of the present invention provides for the actual execution of the test cycles TC(z) by the agent units 130(i) by running the corresponding scripts. Each agent unit 130(i) is capable of carrying out the tests autonomously, i.e., without the need of receiving further data/instructions from the system. Indeed, each agent unit 130(i) carries out the test by running the scripts that were loaded in the previous phase.

(85) However, according to an embodiment of the present invention it is always possible to vary the tasks assigned to the agents 130(i) at any time, by performing again the phase 220 and/or the phase 230 of the procedure 200. In these cases, new scripts are generated. Each agent unit 130(i) is configured to periodically access the test and configuration repository 160 for checking whether new configurations are available.

(86) During the execution of the tests, the agent units 130(i) carry out measures on parameters (e.g., bandwidth, jitter, delay, response time) of the received multimedia content stream 115, acquiring corresponding measured values. Subject to the correct identity verification of the agent unit 130(i) that has submitted the measured values, the measured values are acquired by the acquisition unit 150 through the monitoring network 145 and stored in the measured values repository unit 155.

(87) According to an embodiment of the present invention, the measured values stored in the measured values repository unit 155 can be consulted in form of reports.

(88) The processing unit 152 processes the acquired measured values acquired by the acquisition unit 150 for calculating corresponding QoE indicators indicative of the QoE as perceived by the user enjoying the multimedia content.

(89) According to an embodiment of the present invention, the central unit 140 is further configured to store in the repository unit 155 a so-called agent correct operation table which comprises time indications relating to: the last data transmission carried out by an agent unit 130(i); the new data transmission awaited by an agent unit 130(i); the last configuration downloaded; the last task wait signal by an agent unit 130(i).

(90) In case the agent unit 130(i) is not able to contact the central unit 140, the measured values are temporally stored locally, and provided to the central unit 140 once the contact is reestablished.

(91) When an agent unit 130(i) is communicating with the central unit 140, the former does not carry out any measure.

(92) The last phase 250 of the procedure 200 according to an embodiment of the present invention is directed to verify the correct operation of the agent units 130(i) and the coherence of the acquired measured values.

(93) Regarding the first aspect, according to an embodiment of the present invention, the central unit 140 is configured to automatically check whether each agent unit 130(i) is correctly carrying out its assigned task or not, and if some agent unit 130(i) is in an error situation which cannot enable the correct dialog with the central unit 140 itself. Knowing whether each agent unit 130(i) is correctly carrying out its assigned task or not is a very important issue, in order to be certain that the collected measure values correspond to the desired user behavior model.

(94) These operations are carried out starting from the knowledge of the following information (already generated starting from phase 220) for each test typology Ttype(z): maximum test duration; maximum duration MT(z) of each test cycle TC(z); calendars of the tasks assigned to each agent unit 130(i); information included in the agent correct operation table.

(95) According to an embodiment of the invention, the check is automatically made for each monitoring unit 130(i) by comparing the current time (i.e., the current time stamp) with the maximum duration MT(z) of the test cycle TC(z) carried out from the monitoring unit 130(i), plus a fixed time required to the transmission of new data.

(96) When an error situation relating to an agent unit 130(i) is asserted through said check, the central unit 140 is configured to employ different error management policies, such as for example sending an error email to an operator, or carrying out a ping operation toward the agent unit 130(i) through the monitoring network 145.

(97) Examples of possible situations involving an agent unit 130(i) or the communication between an agent unit 130(i) and the central unit 140 are listed in the following. An agent unit 130(i) succeeds in accomplishing the test and communicating with the central unit 140 within the maximum duration MT(z). In this situation, no error is detected. An agent unit 130(i) succeeds in accomplishing the test within the maximum duration MT(z), but it fails in communicating with the central unit 140. In this situation, the measured values are kept by the agent unit 130(i) for being retransmitted to the central unit 140. If the maximum duration MT(z) time plus a fixed time required to the transmission is expired, the server unit 140 sends an alarm. An agent unit 130(i) fails in accomplishing the test within the maximum duration MT(z) and blocks itself. In this situation, since the agent unit 130(i) is provided with a control system knowing the maximum duration MT(z), after the expiration of this time, the agent unit 130(i) automatically reboots itself and passes to the following test. An agent unit 130(i) fails in accomplishing the test within the maximum duration MT(z) but responds to the control system ping. In this situation, the test is failed with a timeout error, and the agent unit 130(i) passes to the following test.

(98) According to an embodiment of the present invention, during phase 250, the central unit 140 is further configured to verify the coherence of the measured values acquired by the agent units 130(i).

(99) For this purpose, according to an embodiment of the invention, the central unit 140 generates from the configuration table CT outputted by the test scheduler unit 153 and from the calendars outputted by the agent configuration unit 154, a multi-dimensional table, referred to as completion status table, whose data provide an indication about the correct or incorrect execution status of the single tests carried out by the single agent units 130(i).

(100) The completion status table is used to verify the completion status of the data stored in the test repository unit 160. Such verification is used both for reporting the measures and for generating possible critical situation signaling.

(101) For example, it is possible to create a report for allowing a user of the system 125 to consult the completion status table, indicating whether the number of relates to the expected sample survey or to a subset thereof. This report may comprise data aggregated according to selected dimensions that may correspond to the ones selected in phases 210-220 of the procedure, or that may be a subset thereof.

(102) Moreover, a set of critical situation signaling rules may be applied to the data contained in the completion status table in order to point out possible critical situations with respect to the user behavior model implemented by the system 125.

(103) A first exemplary report, which relates to the completion status table of tests according to the dimension values Roma, Milano, Napoli, Firenze, Torino of a dimension City, is disclosed herein below:

(104) TABLE-US-00004 Roma 97% Milano 82% Napoli 89% Firenze 75% Torino 67%
wherein for each dimension value it is reported the percentage of collected measured values with respect to the total amount. An exemplary critical situation signaling rule may provide for having a percentage of collected measured values lower than 80% of the expected total amount. In this example, according to said rule, the cities of Firenze and Torino corresponds to critical situations to be signaled.

(105) A second exemplary report, which relates to the completion status table of tests according to the dimension values Device 1, Device 2, Device 3 of a dimension Device, and for the dimension values Morning, Afternoon, Evening, Night of a dimension Time slot, is disclosed herein below:

(106) TABLE-US-00005 Device 1 Device 2 Device 3 Morning 90% 79% 100% Afternoon 43% 92% 99% Evening 100% 100% 56% Night 100% 86% 77%
wherein for each combination of Device/Time slot dimension values it is reported the percentage of collected measured values with respect to the total amount. An exemplary critical situation signaling rule may provide for having a percentage of collected measured values lower than 75% of the expected total amount. In this example, according to said rule, the combinations Device 1/Afternoon and Device 3/Evening correspond to critical situations to be signaled.

(107) The previous description presents and discusses in detail several embodiments of the present invention; nevertheless, several changes to the described embodiments, as well as different invention embodiments are possible, without departing from the scope defined by the appended claims.