Multimedia resource synchronous pushing method based on heterogeneous network

Abstract

The present disclosure discloses a method for synchronously pushing a multimedia resource based on a heterogeneous network. The method includes: step 1: obtaining, by a server, end-to-end broadband network delay information or broadband network delay and available bandwidth information of different users; step 2: after receiving a client request, calculating, by the server according to the end-to-end broadband network delay information or the broadband network delay and available bandwidth information of the different users, a moment at which the multimedia resource can be synchronously presented; and step 3: sending, by the server, the multimedia resource after the moment at which the multimedia resource can be synchronously presented, and synchronously playing, by a client, content according to a start presentation time of the multimedia resource, so that broadband content of the different users can be synchronously presented.

Claims

1. A method for synchronously pushing a multimedia resource based on a heterogeneous network, wherein the method is any one of the following two methods: method 1: comprising the following steps: step 1, obtaining, by a server, an end-to-end broadband network delay of different users; step 2: after receiving a client request, calculating, by the server according to the end-to-end broadband network delay of the different users, a moment t.sub.tmp at which the client receives a first independent decodable media unit (MU); and step 3: sending at a to moment, by the server, the multimedia resource at a t.sub.ini moment, which is a start moment of a first complete independent decodable media unit MU after the moment t.sub.tmp, at which the multimedia resource is synchronously presented, and synchronously playing, by the client, content according to a start presentation time of the multimedia resource, at and after the t.sub.ini moment, so that broadband content of the different users is synchronously presented, wherein in the step 2 of the method 1, when the end-to-end broadband network delay Δt of the different users is different, the server sends, at the t.sub.0 moment, another multimedia resource at the t.sub.ini moment to ensure synchronization, wherein the t.sub.ini moment is a start moment of a first complete independent decodable media unit (MU) after the moment t.sub.tmp, wherein
0≤t.sub.ini−t.sub.tmp<MU_duration
t.sub.tmp=t.sub.0+Δt in the formula, t.sub.0 is a sending moment, MU_duration is media unit duration, Δt is the end-to-end broadband network delay, and t.sub.tmp is a moment that is obtained by calculation and at which the client receives a first independent decodable media unit MU; method 2: comprising the following steps: step 1: obtaining, by a server, a client request, a broadband network delay Δt and available bandwidth of different users, and duration of an independent decodable media unit MU (media unit) MU_duration; step 2: after receiving the client request, calculating, by the server according to the broadband network delay Δt and the available bandwidth of the different users, a moment t.sub.tmp at which the client receives a first independent decodable media unit (MU); and step 3: sending at a t.sub.0 moment, by the server, the multimedia resource at a t.sub.ini moment, which is a start moment of a first complete independent decodable media unit MU after the moment t.sub.tmp, at which the multimedia resource is synchronously presented, and notifying a client of a play start time of content of the multimedia resource, and synchronously presenting, by different clients, broadband content according to the play start time at and after the t.sub.ini moment.

2. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein a value of MU_duration is: 0 s to 10 s; and a value of Δt is: 0 to ∞.

3. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 2, wherein the value of MU_duration is: 0.5 s; and the value of Δt is: 2 s.

4. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 3, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

5. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 2, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

6. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein in the step 3 of the method 1, the content, which is initially sent by the server is the multimedia resource after the t.sub.ini moment, and to enable a broadband and a broadcast to be synchronous, after receiving the first complete MU by using the broadband, the client does not immediately play the first MU, but instead, plays, at the t.sub.ini moment, the multimedia resource at this moment.

7. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 6, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

8. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein in the method 1, the server sends the multimedia resource to the client, and the client synchronously plays the content according to the start presentation time of the multimedia resource.

9. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 8, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

10. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein in the step 2 of the method 2, when the broadband network delay Δt and the available bandwidth of the different users are different, the server sends, at the t.sub.0 moment, another multimedia resource at the t.sub.ini moment to ensure synchronization, wherein the t.sub.ini moment is a start moment of a first complete independent decodable media unit MU after t.sub.tmp, wherein
0≤t.sub.ini−t.sub.tmp<MU_duration
MU_size=MU_duration*B.sub.r
t.sub.tmp=t.sub.0+MU_size/B.sub.b+Δt in the formula, t.sub.0 is a sending moment, MU_size is an average size of sent MUs, MU_duration is media unit duration, B.sub.r is a constant bit rate at which the multimedia resource is played, B.sub.b is an available bandwidth of a broadband network, Δt is the broadband network delay, and t.sub.tmp is a moment that is obtained by calculation and at which the client receives a first independent decodable media unit MU.

11. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 10, wherein a value of MU_size is: 0 Mbits to 100 Mbits; a value of MU_duration is: 0 s to 10 s; a value of B.sub.r is: 0 Mbps to 50 Mbps; a value of B.sub.b is: 0 Tbps to 1 Tbps; and a value of Δt is: 0 to ∞.

12. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 11, wherein the value of MU_size is: 30 Mbits; the value of MU_duration is: 0.5 s; the value of B.sub.r is: 25 Mbps; the value of B.sub.b is: 1 Mbps; and the value of Δt is: 2 s.

13. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 12, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

14. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 11, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

15. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 10, wherein in the step 3 of the method 2, the content, which is initially sent by the server is the multimedia resource after the t.sub.ini moment, and to enable a broadband and a broadcast to be synchronous, after receiving the first complete MU by using the broadband, the client does not immediately play the first MU, but instead, plays, at the t.sub.ini moment, the multimedia resource at this moment.

16. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 10, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

17. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein in the method 2, when sending the multimedia resource, the server sends a piece of downlink signaling to notify the client of a sequence number of a first sent independent decodable media resource to notify a user of a play time.

18. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

19. The method for synchronously pushing the multimedia resource based on the heterogeneous network according to claim 1, wherein either the method 1 or the method 2 is applicable to a case in which at a particular moment, the different users in the heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for another multimedia resource due to network deterioration, to make the content of the multimedia resources synchronously presented.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The accompanying drawings illustrate one or more embodiments of the present invention and, together with the written description, serve to explain the principles of the invention. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like elements of an embodiment.

(2) FIG. 1 is a schematic timing diagram of a first embodiment of the present disclosure;

(3) FIG. 2 is a schematic timing diagram of a second embodiment of the present disclosure;

(4) FIG. 3 is a flowchart illustrating an example method, according to the first embodiment of the present disclosure;

(5) FIG. 4 is a flowchart illustrating another example method according to the second embodiment of the present disclosure; and

(6) FIG. 5 is an illustration of an example computing device in which the present methods and systems can operate.

DETAILED DESCRIPTION

(7) The following provides preferred embodiments of the present disclosure with reference to the accompanying drawings, to describe the technical solutions of the present disclosure in detail.

(8) Nowadays, diversified terminal presentation manners based on heterogeneous networks have become a trend of development. While watching high-quality broadcast video programs, people are increasing demands for diversified network media services. Usually, media content from a broadcast channel has a small and fixed delay, and therefore it has little effect on the synchronization of media content. Media content from a broadband, for example, audios and videos, subtitles, and multimedia applications is susceptible to the effect of a current IP network, resulting in large and jittery delays, causing problems to content synchronization.

(9) To resolve the foregoing technical problem, implementation details of the technical solutions are described in detail below by using two embodiments of the present disclosure, to facilitate understanding of a person skilled in the art. According to different scenarios, the method of any one of the following two embodiments may be adopted. If a client can learn of end-to-end network delay information, the method in Embodiment 1 is adopted, and if the client can learn of only a broadband network delay not including the time of a process when the media resource enters the client, the method in Embodiment 2 is adopted.

Embodiment 1

(10) The overall technical line of this embodiment is: to learn of end-to-end broadband network delay information of different users by using a corresponding method in a network, for changing a start time at which a server starts to transmit a broadband resource, thereby implementing synchronization of a multimedia resource in a heterogeneous network. The illustrations of this embodiment are shown in FIG. 1 and FIG. 3.

(11) Specifically, method steps of this embodiment are shown in FIG. 3, and the steps include:

(12) first, at step 302, a server may learn of end-to-end broadband network delay information of different users according to some methods, for example, by using signaling information fed back by a client, or by means of estimation measurement;

(13) secondly, at step 304, for a case in which at a particular moment, different users in a heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for a multimedia resource due to network condition deterioration, to make content of the multimedia resources synchronously presented, after receiving a user request, the server calculates, according to network delay conditions of different users, a moment at which the multimedia resource can be synchronously presented; and finally, at step 306, the server sends the multimedia resource after the moment at which the multimedia resource can be synchronously presented, and the client synchronously plays content according to a play start time of the media content, so that broadband part content of the different users can be synchronously presented.

(14) Referring to FIG. 1, in a specific embodiment, the specific implementation is as follows:

(15) step 1: a server receives a client request at a to moment, a downlink end-to-end broadband network delay of different user groups is Δt, and duration of an independent decodable media unit MU (media unit) is MU_duration;

(16) step 2: when end-to-end broadband network delays of the users are different, the server sends, at a t.sub.0 moment, a multimedia resource at a t.sub.ini moment to ensure synchronization, where t.sub.ini is a start moment of a first complete MU after t.sub.tmp, and as shown in formula (1) and formula (2):
0≤t.sub.ini−t.sub.tmp<MU_duration  (1)
t.sub.tmp=t.sub.0+Δt  (2)

(17) In formula (1), MU_duration is media unit duration. In formula (2), t.sub.0 is a sending moment, Δt is a downlink end-to-end broadband network delay, and t.sub.tmp is a moment that is obtained by calculation and at which the client receives the first MU.

(18) In a preferred implementation, the foregoing parameters can select the following values:

(19) MU_duration: 0 s to 10 s, preferably, 0.5 s; and

(20) Δt: 0 to ∞, preferably, 2 s; and

(21) step 3: because initially sent content is the multimedia source at and after the t.sub.ini moment, and to enable a broadband and a broadcast to be synchronous, after a client receives the first complete MU through the broadband, the first MU is not immediately played, but instead, the media content at this moment is played at and after the t.sub.ini moment.

(22) A specific example of multimedia resource synchronization is provided below (a manner of a client request and a manner of notifying the client by the server are not limited, and a signaling manner is used as an example below, and signaling is based on the MMT protocol).

(23) When a user is watching an Olympic live program, because there are a large number of excellent performances, and people have different preferences, the program party specially shoots videos in a plurality of view angles for audience to select.

(24) At this moment, when the audience watches a main program, the audience also watches a close-up performance of a particular athlete in a small image. However, the game of the athlete is completed soon, and the audience wants to immediately change an angle to watch the posture of another athlete. Then the audience sends a request to the server. The request signaling should include an end-to-end network delay fixed_end_to_end_delay. This field can be obtained in HRBM_message, as shown in the following Table 1.1:

(25) TABLE-US-00001 TABLE 1.1 Syntax Values No. of bits Mnemonic HRBM ( ){ message_id 16 version 8 length 16 extension { extension_fields_Byte } message_payload{ max_buffer_size 32 fixed_end_to_end_delay 32 max_transmission_delay 32 } } message_id—message identifier of HRBM version—message version of HRBM length—message length of HRBM extension_fields_Byte—extension field of HRBM max_buffer_size—maximum buffer size of HRBM fixed_end_to_end_delay—end-to-end network delay max_transmission_delay—maximum transmission delay

(26) At t.sub.0=18:00:00, the server receives the request, where fixed_end_to_end_delay=Δt=2 s. Duration of an independent decodable media unit can be obtained from a video resource, where MU_duration=0.5 s.
Δt=2 s
t.sub.tmp=t.sub.0+Δt=18:00:02
t.sub.ini=18:00:022

(27) The server sends, at t.sub.0=18:00:00, a resource of t.sub.ini=18:00:022, and a time at which the client receives a first media resource is 18:00:02, and according to timestamp information timestamp equal to 18:00:022 of the received first media resource, a current media unit is buffered, and the resource is played at 18:00:022.

(28) In this way, a user can watch the posture of another athlete in a different view angle in synchronization between a broadcast and a video on-demand, so as to well resolve the problem that user media resources of broadcast and broadband are asynchronous caused by different network conditions in heterogeneous network transmission.

Embodiment 2

(29) The overall technical line used this embodiment is: to obtain information such as broadband network delays and available bandwidths of different users by using a corresponding method in a network, for changing a start time at which a server starts to transmit a broadband resource, thereby implementing synchronization of a multimedia resource in a heterogeneous network. The illustrations of this embodiment are shown in FIG. 2 and FIG. 4.

(30) Specifically, method steps of this embodiment are shown in FIG. 4, and the steps include:

(31) first, at step 402, a server may obtain information such as broadband network delays and available bandwidths of different users according to some methods, for example, by using signaling information fed back by a client, or by means of estimation measurement;

(32) secondly, at step 404, for a case in which at a particular moment, different users in a heterogeneous network media service request the server for a same multimedia resource or switch different multimedia resources or re-request for a multimedia resource due to network condition deterioration, to make content of the multimedia resources synchronously presented, after receiving a user request, the server calculates, according to network delay conditions of different users, a moment at which the multimedia resource can be synchronously presented; and

(33) finally, at step 406, the server sends the resource after the moment at which the resource can be synchronously presented, notifies the client of a play start time of the media content, so that broadband part content of the different users can be synchronously presented.

(34) Referring to FIG. 2, in a specific embodiment, the specific implementation is as follows:

(35) step 1: a server receives a client request at a to moment, a downlink broadband network delay of different user groups is Δt, an available bandwidth is B.sub.b, a video bit rate is B.sub.r, and duration of an independent decodable media unit MU is MU_duration;

(36) step 2: when the broadband network delays and the available bandwidths of the users are different, the server sends, at a to moment, a multimedia resource at a t.sub.ini moment to ensure synchronization, where t.sub.ini is a start moment of a first complete MU after t.sub.tmp, and as shown in formula (1), formula (2), and formula (3);
0≤t.sub.ini−t.sub.tmp<MU_duration  (1)
MU_size=MU_duration*B.sub.r  (2)
t.sub.tmp=t.sub.0+MU_size/B.sub.b+Δt  (3)

(37) In formula (3), to is a sending moment, MU_size is an average size of sent MUs, and see formula (2) for specific calculation, MU_duration is media unit duration, B.sub.r is a constant bit rate at which the multimedia resource is played, B.sub.b is an available bandwidth of a broadband network, Δt is a downlink broadband network delay, and t.sub.tmp is a moment that is obtained by calculation and at which the client receives the first MU.

(38) In a preferred implementation, the foregoing parameters can select the following values:

(39) MU_size: 0 Mbits to 100 Mbits, preferably, 30 Mbits;

(40) MU_duration: 0 s to 10 s, preferably, 0.5 s;

(41) B.sub.r: 0 Mbps to 50 Mbps, preferably, 25 Mbps;

(42) B.sub.b: 0 Tbps to 1 Tbps, preferably, 1 Mbps; and

(43) Δt: 0 to ∞, preferably, 2 s; and

(44) step 3: because initially sent content is the multimedia source at and after the t.sub.ini moment, and to enable a broadband and a broadcast to be synchronous, after a client receives the first complete MU through the broadband, the first MU is not immediately played, but instead, the media content at this moment is played at and after the t.sub.ini moment.

(45) A specific example of multimedia resource synchronization is provided below. A manner of a client request and a manner of notifying the client by the server are not limited in this embodiment. A signaling manner is used as an example below, and signaling is based on the MMT protocol. Certainly, in other embodiments, the signaling may also be applicable to other media protocols, which are not limited to the MMT protocol.

(46) When a user is watching an Olympic live program, because there are a large number of excellent performances, and people have different preferences, the program party specially shoots videos in a plurality of view angles for audience to select.

(47) At this moment, when the audience watches a main program, the audience also watches a close-up performance of a particular athlete in a small image. However, the game of the athlete is completed soon, and the audience wants to immediately change an angle to watch the posture of another athlete. Then the audience sends a request to the server. The request signaling should include a network delay network_delay and available bandwidth information network_bandwidth. The request message request_message is shown in the following Table 2.1:

(48) TABLE-US-00002 TABLE 2.1 Syntax Value No. of bits Mnemonic request_message( ) { message_id 8 uimsbf version 8 uimsbf length 16 uimsbf message_payload( ) { network_delay 16 uimsbf network_bandwidth 32 uimsbf } }

(49) At t.sub.0=18:00:00, the server receives the request, where Δt=2 s and B.sub.b=512 Kbit/s. The video bit rate and duration of an independent decodable media unit can be obtained from a video resource, where B.sub.r=16482 Kbit/s and MU_duration=0.5 s.
MU_size=MU_duration*B.sub.r=0.5*16482=8241 bits
ΔT=MU_size/B.sub.b+Δt=8241/512+2=16.10 s
t.sub.tmp=t.sub.0+ΔT=18:00:1610
t.sub.ini=18:00:1641

(50) The server sends, at a t.sub.0=18:00:00 moment, a resource of t.sub.ini=18:00:1641, and when sending the resource, the server needs to send a piece of downlink signaling to notify the client of a sequence number of a first sent independent decodable media resource to notify a user of a play time. A response message response_message is as follows:

(51) TABLE-US-00003 TABLE 2.2 Syntax Value No. of bits Mnemonic response_message( ) { message_id 8 uimsbf version 8 uimsbf length 16 uimsbf message_payload( ) { number_of_assets N 16 uimsbf for(i=0; i<N; i++) 32 uimsbf { asset_id 16 uimsbf mpu_sequence_number 32 uimsbf } } }

(52) The client looks up in the table according to a first media resource sequence known in the signaling, to obtain time information timestamp=18:00:1641 and the size MU_size=33 Mbits, and that a moment at which the current client receives the resource is 18:00:1610, buffers the current media unit, and plays the resource at 18:00:1641.

(53) In this way, a user can watch the posture of another athlete in a different view angle in synchronization between a broadcast and a video on-demand, to resolve the problem that user media resources of broadcast and broadband are asynchronous caused by different network conditions in heterogeneous network transmission.

(54) Referring to FIG. 5, the methods and systems of the present disclosure can be implemented on one or more computers. Specifically, a server and any client device can be one or more computers or remote computing devices in FIG. 5. The methods and systems disclosed can utilize one or more computers to perform one or more functions in one or more locations. The processing of the disclosed methods and systems can also be performed by software components. The disclosed systems and methods can be described in the general context of computer-executable instructions such as program modules, being executed by one or more computers or devices. For example, the server includes a bandwidth data processing module configured to obtain broadband network delay and available bandwidth information for different users at respective remote locations. As another example, the server also includes a synchronization module configured to calculate a moment at which the multimedia resource can be is synchronously presented according to the broadband network delay and available bandwidth information of the different users. As another example, the server also includes a notification module configured to notify a client of a play start time of content of the multimedia resource after the moment at which the multimedia resource is synchronously presented. As another example, the server can also include a content delivery module configured to synchronously present broadband content according to the play start time by different clients. The server can be located in the server in a central location, whereas the client devices (remote computing devices) can be located at respective user's location. These program modules can be stored on the mass storage device of the server and one or more client devices. Each of the operating modules can comprise elements of the programming and the data management software.

(55) The components of the server and/or client device can comprise, but are not limited to, one or more processors or processing units, a system memory, a mass storage device, an operating system, a system memory, an Input/output interface, a display device, a display interface, a network adaptor, and a system bus that couples various system components. The server and one or more client devices can be implemented over a wired or wireless network connection at physically separate locations. By way of example, a remote client device can be a personal computer, portable computer, smartphone, a network computer, a peer device, or other common network node, and so on. Logical connections between the server and one or more client devices can be made via a network, such as a local area network (LAN) and/or a general wide area network (WAN).

(56) The foregoing specific embodiments further describe, in detail, the technical problem resolved by the present disclosure, and the technical solutions and beneficial effects of the present disclosure. It should be understood that the foregoing descriptions are merely specific embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the present disclosure should fall within the protection scope of the present disclosure.