PREDICTING MULTIMEDIA SESSION MOS

Abstract

It is provided a method, performed by a MOS, Mean Opinion Score, estimator, for predicting a multimedia session MOS. The multimedia comprises a video and an audio, wherein video quality is represented by a list of per time unit scores of a video quality, an initial buffering event and rebuffering events in the video, and wherein audio quality is represented by a list of per time unit scores of audio quality. The method comprises: generating video features from the list of per time unit scores of the video quality; generating audio features from the list of per time unit scores of the audio quality; generating buffering features from the initial buffering event and rebuffering events in the video; and estimating a multimedia session MOS from the generated video features, generated audio features and generated buffering features by using machine learning technique.

Claims

1. A method, performed by a MOS, Mean Opinion Score, estimator, for predicting a multimedia session MOS, wherein the multimedia comprises a video and an audio, wherein video quality is represented by a list of per time unit scores of the video quality, an initial buffering event and rebuffering events in the video, and wherein audio quality is represented by a list of per time unit scores of the audio quality, the method comprising: generating one or more of the group consisting of: video features from the list of per time unit scores of the video quality; audio features from the list of per time unit scores of the audio quality; buffering features from the initial buffering event and the rebuffering events in the video; and estimating the multimedia session MOS from the one or more of the generated video features, generated audio features, and generated buffering features by using machine learning technique.

2. The method according to claim 1, wherein the video features comprise a feature being a first percentile of the per unit time scores of the video quality.

3. The method according to claim 1, wherein the video features comprise a feature being a fifth percentile of the per unit time scores of the video quality.

4. The method according to claim 1, wherein the video features comprise a feature being a fifteenth percentile of the per unit time scores of the video quality.

5. The method according to claim 1, wherein the step of estimating is based on a random forest based model.

6. The method according to claim 1, wherein the buffering features comprise a feature being total buffering time.

7. The method according to claim 1, wherein the buffering features comprise a feature being number of the rebuffering events.

8. The method according to claim 1, wherein the buffering features comprise a feature being percentage of buffering time divided by video time.

9. The method according to claim 1, wherein the buffering features comprise a feature being number of the rebuffering events per video length.

10. The method according to claim 1, wherein the buffering features comprise a feature being last seen rebuffering from the end of the video.

11. A MOS, Mean Opinion Score, estimator for predicting a multimedia session MOS, wherein the multimedia comprises a video and an audio, wherein video quality is represented by a list of per time unit scores of the video quality and an initial buffering event and rebuffering events in the video and wherein audio quality is represented by a list of per time unit scores of the audio quality, the MOS estimator comprising processing means and a memory comprising instructions which, when executed by the processing means, causes the MOS estimator to: generate one or more of the group consisting of: video features from the input list of per time unit scores of the video quality; audio features from the input list of per time unit scores of the audio quality; buffering features from the initial buffering event and the rebuffering events in the video; and estimate the multimedia session MOS from the one or more of the generated video features, generated audio features, and generated buffering features by using machine learning technique.

12. The MOS estimator according to claim 11, wherein the video features comprise a feature being a first percentile of the per unit time scores of the video quality.

13. The MOS estimator according to claim 11, wherein the video features comprise a feature being a fifth percentile of the per unit time scores of the video quality.

14. The MOS estimator according to claim 11, wherein the video features comprise a feature being a fifteenth percentile of the per unit time scores of the video quality.

15. The MOS estimator according to claim 11, wherein the instructions to estimate comprise instructions which, when executed by the processing means, causes the MOS estimator to estimate using a random forest based model.

16. The MOS estimator according to claim 11, wherein the buffering features comprise a feature being total buffering time.

17. The MOS estimator according to claim 11, wherein the buffering features comprise a feature being number of rebuffering events.

18. The MOS estimator according to claim 11, wherein the buffering features comprise a feature being percentage of buffering time divided by video time.

19. The MOS estimator according to claim 11, wherein the buffering features comprise a feature being number of rebuffering events per video length.

20. The MOS estimator according to claim 11, wherein the buffering features comprise a feature being last seen rebuffering from the end of the video.

21. A MOS, Mean Opinion Score, estimator comprising: a generating module, configured to generate video features from an input list of per time unit scores of video quality, generate audio features from the input list of per time unit scores of audio quality and generate buffering features from an initial buffering event and rebuffering events in a video; and a predicting module, configured to predict a multimedia session MOS from the generated video features, generated audio features and generated buffering features by using machine learning technique.

22. A non-transitory computer-readable storage medium comprising a computer program product including instructions to cause at least one processor to: generate video features from an input list of per time unit scores of video quality; generate audio features from the input list of per time unit scores of audio quality; generate buffering features from an initial buffering event and rebuffering events in the video; and estimate a multimedia session MOS, Mean Opinion Score, from the generated video features, generated audio features and generated buffering features by using machine learning technique.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The invention is now described, by way of example, with reference to the accompanying drawings, in which:

[0039] FIG. 1 illustrates a system architecture according to the embodiments presented herein;

[0040] FIGS. 2A-C are schematic graphs illustrating buffering and bitrate over time;

[0041] FIG. 3 shows the aggregation module of FIG. 1 according to embodiments presented herein;

[0042] FIG. 4 depicts a random forest model according to one scenario which can be applied in embodiments presented herein;

[0043] FIG. 5 illustrates the steps performed by a MOS estimator according to the embodiments presented herein;

[0044] FIG. 6 illustrates a MOS estimator according to the embodiments presented herein;

[0045] FIG. 7 is a schematic diagram illustrating the relationship between transmitter and receiver;

[0046] FIG. 8 is a schematic diagram illustrating components of the MOS estimator of FIG. 2; and

[0047] FIG. 9 shows one example of a computer program product comprising computer readable means.

DETAILED DESCRIPTION

[0048] The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0049] The subjective MOS is how humans rate the quality of a multimedia sequence. Objective MOS estimation is using models to predict/estimate how humans will rate it. In general, parametric based methods are usually used to predict the multimedia MOS. This kind of parametric based methods usually results in quite a large prediction error.

[0050] The basic idea of embodiments presented herein is to predict the multimedia session MOS using machine learning approaches. The input for the prediction includes the following:

[0051] 1. A list of per time unit scores of the video quality

[0052] 2. A list of per time unit scores of the audio quality

[0053] 3. The initial buffering event and rebuffering events in the video.

[0054] A time unit may be a second. Thus, the lists of per time unit scores of the video and audio quality may be obtained per second.

[0055] From these inputs, a number of features are generated. Next, using these features, a machine learning model is trained with random forest to predict session MOS. Each feature is such that a Boolean condition can be obtained by evaluating the feature according to some criteria, e.g. comparing a scalar feature with a certain value.

[0056] FIG. 1 illustrates the system architecture according to the embodiments presented herein. FIG. 1 relate to a receiver (see 13 of FIG. 7) of a multimedia session. We have a video module 21 to predict the per time unit (e.g. second) second scores of the video quality of a video sequence. The input 15 to this video module 21 includes video bitrates, codecs, screen size, frame types, etc. The output 25 of this video module 21 is a list of scores for video quality, with each element representing the quality for one second. For instance, the video output 25 includes a series of video scores such as [3.1, 3.2, 3.3, 3.3, 3.5, . . . ].

[0057] We also have an audio quality module 22 to predict the per time unit (e.g., second) scores of the audio quality of a video sequence. The input 16 to this audio module 22 includes audio bitrates, audio codecs, etc. The output 26 is also a list of scores, but for audio quality. For instance, the audio output 26 includes a series of audio scores such as [2.1, 2.2, 3.3, 3.3, 3.5, . . . ].

[0058] Also, there is a buffering module 23 to provide statistics of the buffering during this video playout. The input 17 to the buffering module is buffering events 17. The output 27 of the buffering module 23 contains a list of buffering events, where each event includes the time since start of video clip, and the duration of buffering. For instance, the buffer output 27 includes a series of buffering events such as [[0, 2], [10, 1]], where the first parameter of each event is the video time (i.e. a timestamp in a media timeline) at which point the buffering started and the latter is called buffering time (i.e. duration of the buffering). If the video time is 0, it is called the initial buffering. Otherwise, it is considered as the rebuffering.

[0059] The video input 15, the audio input 16 and the buffering input 17 all relate to one multimedia session, i.e. a single reception of multimedia comprising both video and audio.

[0060] An aggregation module 1, takes the outputs 21, 22, 23 from the video module 21, audio module 22, and buffering module 23 to predict the final session MOS score 29. The aggregation module 1 is also referred to as a MOS estimator herein.

[0061] FIGS. 2A-C are schematic graphs illustrating buffering. Time progresses along the horizontal axis. The vertical axis represents bitrate.

[0062] One impact in quality is rebuffering (when the transmission speed is not high enough), as seen in FIG. 2A. This can also be referred to as stalling. There are there three gaps (with zero bitrate) which represents rebuffering.

[0063] When the transmission capacity in a network fluctuates, for instance for a wireless connection, the media player (in the receiver, 13 of FIG. 7) can often select to adapt the bitrate, so that the video can still be delivered, albeit with sometimes worse quality (lower bitrate, lower resolution etc.). FIG. 2B illustrates an example for a 60-second video, where the height represents the bitrate, and each block is 5 seconds long. The quality models of the prior art do not account for this adaptation of bitrate.

[0064] Note that also in the case of adaptive bitrate, rebufferings may occur, so the combinations of adaptations and rebufferings need to be handled, as in the more complex example of FIG. 2C.

[0065] FIG. 3 shows the aggregation module 1 of FIG. 1 according to embodiments presented herein.

[0066] The aggregation module 1 here uses an objective model for estimation of multimedia streaming quality, belonging to the No-Reference context.

[0067] The aggregation module 1 contains several sub-modules. First, there is a video feature generation module 30. Second, there is an audio feature generation module 31. Third, there is a buffering feature generation module 32. Finally, there is a random forest prediction module 35 together with model builder 36.

[0068] The video feature generation module 30 generates features from the list of per time unit video scores 25 (obtained from the video module 21 of FIG. 1). The time unit can be any suitable time unit. For example, a time unit may be a second. There are seven features that are significant for video MOS prediction. Hence, the output of the video feature generation module 30 contains one or more of the following 7 features: [0069] 1. Video length (the length of the media) [0070] 2. 1.sup.st percentile of the per second video score [0071] 3. 5.sup.th percentile of the per second video score [0072] 4. 15.sup.th percentile of the per second video score [0073] 5. Average of the first part (the first ⅓ of the video) of the per second video score [0074] 6. Average of the middle part (the second ⅓ of the video) of the per second video score [0075] 7. Average of the last part (the final ⅓ of the video) of the per second video score

[0076] However, the embodiments presented herein is by no means limited to the seven features identified above, nor to the exact numerical values given above.

[0077] The audio feature generation module 31 is used to generate features from the list of per second audio scores 26 (obtained from the audio module 22 of FIG. 1). It has been found empirically that audio quality is less important for MOS prediction than video quality. Therefore, it is here presented two features which may be used to capture audio quality. The two features are: [0078] 1. Average of the first part (first ½ of the audio) of the per second audio score [0079] 2. Average of the second part (second ½ of the audio) of the per second audio score.

[0080] The buffering feature generation module 32 is used to generate features based on buffering events. The input is the list of buffering events 27 (obtained from the buffering module 23 of FIG. 1), where each event includes a video time, and a buffering time. The video time shows how much (when) in the video playout the buffering started, and the buffering time shows how much time of buffering it is. It is to be noted that the video time does not include the length of any possible previous bufferings. If the video time is o, it means the buffering is initially buffering. Otherwise it is more considered as rebuffering. In other words, the initial buffering (at video time 0) is not a rebuffering but all subsequent bufferings (at video times after 0) are considered rebuffering.

[0081] An example of a list of buffering is shown below. This example comprises three buffering events, one at the beginning of the video with 3 seconds of buffering, one in 10 seconds of the video with 2 seconds of rebuffering, and one in 50 seconds of the video with 1 second of rebuffering.

[0082] The buffering event input 27 are then represented by: [[0, 3], [10, 2], [50, 1]]

[0083] One or more of the following features can be generated according to embodiments herein: [0084] 1. Total buffering time (With some adjustment for initial buffering) [0085] 2. Number of rebuffering events [0086] 3. Percentage of buffering time divided by video time [0087] 4. Number of rebuffering events per video length [0088] 5. Last seen rebuffering from the end of the video

[0089] However, the embodiments presented herein are not limited to the five features above—there may be more features used in this module.

[0090] The total buffering time sums up the seconds of buffering in the video playout. However, the initial buffering event is given ⅓.sup.rd of weight compared to other rebuffering events as users tend to forget about the events occurring at the start of streaming due to the effect of memory. Also, our data driven approach shows that this approach provides higher prediction accuracy. With this approach, the total buffering time is 3/3+2+1=4 on the sample above.

[0091] The number of rebuffering events is the count of all the rebuffering events. In the above example, number of rebuffering events is two. The initial buffering is not considered a rebuffering event.

[0092] The third feature percentage of buffering time compared to video time is calculated by taking the value of feature 1 divided by the total video length. This can also be thought of as a radio of stalling duration. As the test set of video sequences of different length, using this feature is also good. The value of feature 3 in the above example becomes 4/60=0.067 if the media length is 60 seconds long.

[0093] The fourth feature, i.e., the number of rebuffering events per video length is to take feature 2 divided by video length. The value for the example is then 2/60=0.033. This can also be thought of a frequency of rebuffering events.

[0094] Finally, there is a feature about last seen rebuffering. In the example, the video length is 60 seconds, while the (start of the) last rebuffering is done at 50 seconds, so the last seen rebuffering from the end of the video is 60−50=10 seconds. If there is no rebuffering in the session, the last seen rebuffering is set to the media length.

[0095] A random forest MOS prediction module 35 takes in inputs (features) from the video feature generation module 30, audio feature generation module 31 and the buffering feature generation module 32. In the case when the inputs from the modules are as described above, there is a total of fifteen inputs (i.e. features): the seven inputs are from the video feature generation module 30, two inputs are from the audio feature generation module 31 and five inputs are from the buffering feature generation module 32. Finally (and optionally), one more feature about the device type 33, which indicates if the test is on a mobile device or PC device is given as input to the model in the random forest MOS prediction module 35. The output 39 of the random forest MOS prediction module 35 is an estimated (or predicted) MOS value.

[0096] FIG. 4 depicts a random forest model 55 according to one scenario. The random forest model is applied in the random forest MOS prediction module 35 of FIG. 3. It is to be noted that other random forest models with different number of trees and/or depths can be applied. The input to the model 55 is a feature set input with values 50, i.e. one or more of the features described above.

[0097] The random forest based model 55 may be built on a number of trees, each tree characterised with a maximum depth. For a model according to one scenario, the random forest is built on fifty trees. Each tree has a maximum depth of eight, which means that hierarchically there can be only eight levels in the trees. The depth is set to eight, which provides a good trade-off between accuracy and complexity of model. Upon receiving inputs from video, audio and buffering modules for a particular streaming session, it estimates MOS in the following way.

[0098] The model 55 parses through fifty trees 51a-n that were already constructed during a training phase of the model 55. At each node in a tree, there is a condition check on the value of a certain feature (for example, among the fifteen features described above) is smaller than the specified value of the tree 30 node, if the answer to the condition is YES, it proceeds to the left child node. Otherwise, it proceeds to the right child node. It recursively goes through all the levels in a tree, until it reaches a leaf node of the tree, where it gets a MOS estimate. The leaf node can be reached at any depth between one and eight, depending on trees and the values of feature.

[0099] The above process is performed for all the fifty trees, which can be done in parallel for all the trees. Finally, there are fifty estimates of MOS scores for the streaming session obtained from the corresponding fifty trees. The average of the fifty MOS scores is then calculated. Henceforth, the module gives an output (29 of FIG. 3) in the form of average MOS score, which is the final estimate of MOS score for the considered streaming session.

[0100] An example of the random forest model 55 with fifty trees and maximum depth of eight is shown in FIG. 4. The nodes in trees are ovals and leaves are depicted as rectangles. With a given input of feature values to the aggregation module, in the first tree 51a, the leaf node is reached at depth 2 with an output MOS estimate of 4.7 (the leaf being indicated with a solid line bordered rectangle). Similarly, in tree fifty, the MOS estimate for this example is 3.6, which is reached at depth 3 (the leaf being indicated with a solid line bordered rectangle). The paths to leaves in the above examples are shown by thick lined arrows.

[0101] The model in the embodiments presented herein may optionally be extended by introducing any one or more several other features, some of which are listed here: [0102] 1. Mean play duration: the play duration (t.sub.p) is the time interval in which a user enjoys smooth playout of multimedia streaming without any rebufferings. It can also be defined as the time duration between two subsequent rebuffering events and thus expressed as:

t.sub.p=T.sub.r.sup.i+1−T.sub.r.sup.i [0103] where T.sub.r.sup.i is the time when rebuffering event i occurs. [0104] Thus, mean play duration (t.sub.p) is the mean of all the episodes of smooth playouts occurring within a stream. [0105] 2. Mean rebuffering duration: the mean rebuffering duration (t.sub.r) is the mean length of all the individual rebuffering events occurring within a streaming session. [0106] 3. Rebuffering frequency: the rebuffering frequency (f.sub.r) is a measure of repetition of rebuffering events within a streaming session. It can be expressed by:

[00001]$f_r=\frac{1}{{\stackrel{\_}{t}}_p}$ [0107] 4. Mean cycle duration: The mean cycle duration (t.sub.c) is the sum of mean play and rebuffering durations within a streaming session and can be expressed by:

t.sub.c=t.sub.p+t.sub.r [0108] 5. Rebuffering duration difference between the first and the last half of streaming: [0109] Assuming that the amount of impact of total rebuffering duration in the first and the last half of a session differs due to the user memory, a feature based on the difference between rebuffering durations of the first and the last half of streaming session is proposed. The durations of all rebuffering events occurring within each half are added. The difference is then taken by subtracting the sum of duration of rebufferings of first half from the last half. For example, if the total duration of all the rebuffering events occurring within the first 50% and the last 50% parts of a video streaming session is 10 seconds and 15 seconds, respectively, the value of this feature will become 5 seconds. [0110] 6. Rebuffering count difference between the first and the last half: [0111] Like the above feature, this feature is the difference between counts of rebufferings occurring within first and last half of streaming session. [0112] 7. Standard deviation of per-second audio MOS scores: the standard deviation of all the per-second audio MOS scores given by the audio module is calculated. [0113] 8. Minimum per-second video MOS score: the minimum per-second score input from video module within a streaming session. [0114] 9. Maximum per-second video MOS score: the maximum per-second score from video module within a streaming session. [0115] 10. Number of per-second video MOS degradations: the number of degradations recorded in per-second MOS values over time within a streaming session. Each degradation refers to a drop in per-second video MOS value as compared to the previous video MOS value. [0116] 11. Per-second change in video MOS scores: the list of per-second MOS scores are collected from the video module. The difference between every two subsequent per-second video MOS values is calculated. These differences are squared and then their sum is taken, which refers to the per-second change in video MOS scores. The following equation expresses this feature:

μ=Σ.sub.i.sup.n(vMOS.sub.i+1−vMOS.sub.i).sup.2 [0117] where vMOS.sub.i is the video MOS score of the i.sup.th second of a streaming session.

[0118] FIG. 5 is a flow chart illustrating the steps performed by a MOS estimator according to embodiments presented herein. The method is performed for predicting a multimedia session MOS, wherein the multimedia comprises a video and an audio, i.e. a video stream and an audio stream. As explained above, video quality is represented by a list of per time unit scores of a video quality, an initial buffering event and rebuffering events in the video. Also, audio quality is represented by a list of per time unit scores of audio quality.

[0119] In a generate video features step 40, video features are generated from the list of per time unit scores of the video quality. The video features may comprise a feature being a first percentile of the per unit time scores of the video quality. The video features may comprise a feature being a fifth percentile of the per unit time scores of the video quality. The video features may comprise a feature being a fifteenth percentile of the per unit time scores of the video quality.

[0120] In a generate audio features step 42, audio features are generated from the list of per time unit scores of the audio quality.

[0121] In a generate buffering features step 44, buffering features are generated from the initial buffering event and rebuffering events in the video. The buffering features may comprise a feature being total buffering time. The buffering features may comprise a feature being number of rebuffering events. The buffering features may comprise a feature being percentage of buffering time divided by video time. The buffering features may comprise a feature being number of rebuffering events per video length. The buffering features may comprise a feature being last seen rebuffering from the end of the video.

[0122] In a predict a multimedia session MOS step 46, a multimedia session MOS (39 of FIG. 3) is estimated from the generated video features, generated audio features and generated buffering features by using machine learning technique. In one embodiment, the estimating is based on a random forest based model.

[0123] FIG. 6 illustrates a MOS estimator according to the embodiments presented herein, wherein the MOS estimator comprises functional modules. The modules are implemented using software instructions such as a computer program executing in the MOS estimator. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in FIG. 5.

[0124] A generating means 70 corresponds to steps 40, 42 and 44. A predicting means 72 corresponds to step 46.

[0125] FIG. 7 is a schematic diagram illustrating the relationship between transmitter and receiver. A transmitter lo transmits multimedia 11 comprising video and audio, over a channel 12, to the receiver 13. The channel 12 can be based on wireless communication and/or wire-based communication. As explained above, the channel 12 is not perfect (e.g. due to interference, congestion, fading, etc.). Hence, the quality experienced at the receiver 13 can vary and can be expressed by MOS.

[0126] FIG. 8 is a schematic diagram illustrating components of the MOS estimator 1 of FIG. 3. The components can be shared with a host device hosting the MOS estimator. A processor 60 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit etc., capable of executing software instructions 67 stored in a memory 64, which can thus be a computer program product. The processor 60 can be configured to execute the method described with reference to FIG. 5 above.

[0127] The memory 64 can be any combination of read and write memory (RAM) and read only memory (ROM). The memory 64 also comprises persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0128] A data memory 66 is also provided for reading and/or storing data during execution of software instructions in the processor 60. The data memory 66 can be any combination of read and write memory (RAM) and read only memory (ROM).

[0129] The MOS estimator further comprises an I/O interface 62 for communicating with other external entities. Optionally, the I/O interface 62 also includes a user interface.

[0130] Other components of the MOS estimator are omitted in order not to obscure the concepts presented herein.

[0131] FIG. 9 shows one example of a computer program product comprising computer readable means. On this computer readable means a computer program 91 can be stored, which computer program can cause a processor to execute a method according to embodiments described herein. In this lo example, the computer program product is an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product 64 of FIG. 8. While the computer program 91 is here schematically shown as a track on the depicted optical disk, the computer program can be stored in any way which is suitable for the computer program product, such as a removable solid state memory, e.g. a Universal Serial Bus (USB) drive.

[0132] The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.