NETWORK OFFSET

Abstract

There is provided a method in a remote media production system in an IP network, in which media production is performed at a stadium or the like, and the produced media content is transferred to a home studio for final production. The media content is transported in individual data streams over a network. In a receiving node R an aggregate of individual delays for the transferred data streams is monitored and form basis for determining at least one network delay correction factor or at least one common network offset for the individual data streams. The method further comprises time adjusting data transmitted over the network with the common network offset.

Claims

1. A method for remote media production in an IP network comprising: at least one receiving node: monitoring an aggregate of individual delays for a multiple of individual data streams being transmitting over the network from at least one production node to said receiving node and determining at least one common network offset for compensating propagation time of traffic over said individual streams in the IP network based on said aggregate of individual delays; wherein said method further comprises time adjusting data in said data streams transmitted over said network with said at least one common network offset.

2. A method according to claim 1, wherein said individual data streams are associated with at least one of a number of predetermined groups.

3. A method according to claim 1, wherein said step of determining at least one common network offset is at least one of based on time stamps, performed periodically, and performed continuously.

4. A method according to claim 1, wherein said step of determining a common network offset comprises determining in said LSET at least one of an average delay value, a minimum delay value, a maximum delay value, an optimum delay value, and a network delay correction factor within at least one predetermined max and/or min margin value.

5. A method according to claim 4, wherein said optimum delay, and/or margin values are determined by at least one of: calculation, estimation, based on network properties, by a third part, via a management interface, and by machine learning.

6. A method according to claim 1, wherein said step of time adjusting data comprises performing at least one of: timestamping data of said data streams with a time stamp compensated based on the network delay correction factor, and adding an additional time stamp compensated based on said common network offset.

7. A method according to claim 1, wherein said step of time adjusting data comprises at a gateway or said receiving node: buffering data received in said respective individual data streams and forwarding said buffered data at a time compensated with said common network offset.

8. A method according to claim 1, wherein said step of time adjusting data comprises at said production node time stamping data of at least one source device with a local time stamp compensated by said common network offset; and immediately transmitting said data.

9. A method according to claim 1, wherein said step of time adjusting data comprises at said production node adjusting time of a source clock with said common network offset.

10. A method according to claim 9, wherein the adjusted time of the source clock is distributed back over the network using a network time protocol.

11. A method according to claim 9, where a node in said production node is adjusting said source clock using a reference source clock and the common network offset received from said receiving node.

12. A method according to claim 9, wherein said step of time adjusting further comprises adjusting time compensated data for frame start alignment with respect to frame start of locally/studio generated data streams.

13. A node in a distribution network comprising means for performing a method according to claim 9.

14. A software module adapted to perform the method according to claim 9, when executed by a computer processor.

15. A Gateway in a WAN production network comprising means for performing a method according to claim 9.

Description

DRAWINGS

[0039] The above, as well as additional objects, features and advantages of the present invention, will be better understood through the following illustrative and non-limiting detailed description of preferred embodiments of the present invention, with reference to the appended drawings, where the same reference numerals will be used for similar elements, wherein:

[0040] FIG. 1 is a schematic block diagram illustrating a remote to central media production system according to embodiments of the present inventive concept;

[0041] FIG. 2 is a schematic illustration of network delay determined from an aggregate set of individual delays according an embodiment of the present inventive concept;

[0042] FIG. 3 is a schematic illustration of an exemplifying embodiment of the present inventive concept;

[0043] FIG. 4 is a schematic illustration of an exemplifying embodiment of the present inventive concept;

[0044] FIG. 5 is a schematic illustration of an exemplifying embodiment of the present inventive concept;

[0045] FIG. 6 is a schematic illustration of the adjusted reference source clock used to generate and send streams from a remote production site to optimize delay at the receiving node to avoid frame buffering; and

[0046] FIG. 7 is a schematic block diagram illustrating a distributed media production system, where different parts of the production are performed at different sites.

[0047] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0048] FIG. 1 is a block diagram schematically illustrating a remote media production system 100 of IP type for live or prerecorded remote to central production of e.g. a sports event, in view of which aspects of the inventive concept will be described with additional details associated with exemplifying embodiments discussed further below.

[0049] The remote media production system 100 shown in FIG. 1 comprises a production studio 110 and remote production nodes 120, 130, 140, and 150 corresponding to different locations and/or sub-events at one or more venue sites, i.e. a stadium, a local studio and a mobile reporting team stationed outside the stadium (not shown). The remote production nodes 120, 130, 140, and 150 each comprises a plurality of source devices, respectively, e.g. cameras 122, 132, 142, and 152, sound recorders 123, 133, 143, and 153, and processing/transceiver equipment 124, 134, 144, 154 for transmitting data streams comprising video, audio and ancillary data (ANC) from each of the remote productions nodes 120, 130, 140, and 150 over the same or different communication links to the production studio 110 (or Broadcast Centre) over a network 50. The network may be a fixed network (e.g. a LAN, a WAN, the Internet), a wireless network (e.g. a cellular data network), or some combination of these network types, to the receiving node, e.g. a broadcast location such as a local TV studio. The network 50 does not need to be a dedicated network but can be shared with other services. Control instructions and other data may further be transmitted from the production studio 110 over the network to the remote production nodes 120, 130, 140, and 150. The remote production nodes 120, 130, 140, 150 at the venue, optionally an RBC (not shown) and the production studio 110 are thus interconnected with the network 50 that carries all video, audio and data signals. The production studio 110 contains a main portion of production equipment 111 needed to put together a TV-production, like e.g. processing equipment for replays, editing, camera control of cameras at the remote production nodes, audio and video production.

[0050] The event is typically captured by the multiple of cameras 122, 132, 142, and 152, and sound recorders 123, 133, 143, and 153, placed at different locations and sub-events at the event site, and each camera and/or microphone generates an individual IP signal corresponding to its capture of the event including e.g. audio data, video data, metadata etc., the protocol and contents depending on the associated source, which is transported from the one or more productions nodes over the network in links (paths) to the production studio or other receiving node (gateway or processing nodes) as individual data streams.

[0051] In a remote production system as described herein, the source devices, such as the cameras, microphones etc., typically derive a timing reference from a common master source, time reference t.sub.ref, such that the timing of the internal clocks in all the source devices is accurately synchronized. A protocol based on e.g. the Precision Time Protocol (PTP) or GPS etc. can be used for such a clock synchronization.

[0052] Alternatively, generator-locked instruments are used at the production nodes to generate video signals for further transport to the production studio which are then to be syntonized. Syntonized video signals are frequency-locked, but because of delays in the network (caused by e.g. propagation delay due to different path lengths through the network, processing- and queuing delays in routers in the path, and transmission delay at the source node), the synchronized signals will exhibit differing phases at various points in a television system. Modern video equipment, e.g. production switchers that have multiple video inputs may comprise a variable delay on its inputs to compensate for some phase differences and time all the input signals to phase coincidence.

[0053] According to the current inventive concept, the problem with different delay of data streams in the network is attended to by means of the disclosed method for remote media production in an IP network. Monitoring of an aggregate of delays for a set of individual links (LSET) may be carried out in at least one receiving node, here represented by the production studio 110 but which may also refer to a gateway. The receiving node comprises suitable logic, circuitry, interfaces and code capable of monitoring the set of individual data streams and the corresponding delays (LSET) such that a network offset can be determined based on the aggregate of individual delays. Each of the individual delays may be caused by any of or combinations of a non-exhausted list comprising link delay, buffer delay through one or more network nodes, encoding, processing/format conversion (e.g. MPEG-coding conversion) etc. From the aggregate of delays at least one network delay correction factor, CORR factor, is determined, from which at least one CORR factor is selected and utilized for time compensating data transmitted over the individual links. The CORR factor may interchangeable be referred to as a network offset correction factor or a common network offset. Time compensating data is preferably performed by manipulating time stamps, restamping time stamps contained in packets or adding time stamps in packets of the data streams. Optionally, in addition to providing such time compensation or time adjustment, data may be buffered to provide a required alignment of signals being forwarded into the production studio 110.

[0054] In FIG. 2, a number N individual delays of different streams and/or potentially group of streams are presented in a bar chart, which delays are derived by monitoring corresponding media streams being transmitting over a network from at least one production node to the receiving node. It should be realized that grouping of individual delays can be formed from groups coming from the same stadium, but it can also be subgroups within the group that passes another production/processing facility or a cloud/data center.

[0055] The individual delays may be determined based on time stamps included in the individual data streams using the same clock (e.g., via GPS or network synchronization method) used at the receiving side, and may be monitored continuously or periodically, the latter can be advantageous to reduce resource requirements. Each time stamp in the individual data streams represents the value of the reference time t.sub.ref at the moment of transmission/creation of the specific data packet to be transmitted to the receiving node and may be compared to a receive time at the moment of reception of the same packet at the receiving node (on a condition that the receiving node has the same reference time t.sub.ref).

[0056] Depending on where the delay is measured, the delay for specific data streams may include delay caused by data compression, sound processing, and cloud processing. From the monitored aggregate, at least one network offset correction (CORR) factor is determined, e.g. an average delay value (Δaverage), a minimum delay value (Δmin), a maximum delay value (Δmax), an optimum delay value (Δopt), and within a predetermined margin value between a min and max value (or min and max range, respectively), preferably selected on a group basis.

[0057] According to an embodiment of the present inventive concept, from the monitored aggregate, at least one CORR factor is determined on a concept based of at least one group, which group concept is described in further detail herein under. The at least one CORR factor is selected from an average delay value, Δaverage of the group, a minimum delay value, Amin of the group, a maximum delay value, Δmax of the group, an optimum delay value, Δopt of the group, and optionally within a predetermined margin value between a min and max value selected on a group basis.

[0058] The optimum delay, and/or max- and min margin values may be determined by one of calculating or estimating CORR factor values, or as when utilizing margin values needs to be set based on experience. Machine learning may be applied based on analyzing for example packet delay variations in incoming traffic. According to an embodiment of the method, supervised machine learning is utilized to determine an optimum delay, and/or max- and min margin values, by constructing a predictive data analysis algorithm based on a model of the remote production and/or network system and that makes predictions based on evidence, e.g. previously monitored patterns in the LSET for a specific group in the presence of uncertainty. A supervised learning algorithm takes the known set of input data, i.e. previously monitored patterns in the LSET for a specific group and known responses (output) to that input data. The model is then trained to generate reasonable predictions for the response to new data. The algorithm may be designed for estimation of optimum delay and/or min-max margin values.

[0059] Referring again to FIG. 1, consider now that the illustrated media production system 100 covers a big sports event. As previously explained the source devices providing signals from the sports event are here represented by video cameras, sound recorders and the like. A specific set of the source devices, e.g. source devices 132, 133, and 134 at production node 130, are associated with a first group assigned to cover a specific part of the total event, a first sub-event, e.g. to cover a hockey game played in a rink, while other source devices, e.g. source devices 142-144 of production node 140, are associated with a second group dedicated to cover figure skating, i.e. a second sub-event, and yet other source devices, e.g. source devices 122-124 at production node 120, are associated with a third group dedicated to a local event studio. At the same time, video cameras 122 and 132 are associated with a fourth group which represents e.g. a specific camera technology, that is, all or some of source devices associated with a first group and/or all or some of source devices associated with a second group can be associated with e.g. a third group or fourth group. Subgroups of signals/devices covering the same stadium may for instance be processed in a specialized site, like audio processing at a specific geographically separated site and ANC data which is processed automatically in a cloud service where the cloud service location can be located wherever. Then the CORR factor at the home studio needs to be determined considering the different transport delays and processing times of the two sub-groups of the one and same stadium group. That is, the aggregate monitoring of/individual/delays is associated with at least one of several predetermined groups. The predetermined groups are selected from a non-exhaustive list comprising specific production nodes, specific sub-events, type of media stream, such as a video stream, audio stream, metadata/ANC stream, audio-video stream, a specific technology of the production nodes/receiving nodes, and geographic regions. The groups may further be arranged as hierarchical groups, e.g. a group.1 corresponding to a stadium1, group1.1 corresponding to audio from the stadium 1, group1.2 corresponding to UHD from stadium 1, group1.3 corresponding to slow mo from stadium 1, etc.

[0060] Removing Minimum WAN Delay Offset

[0061] With reference now to FIG. 3, a media distribution system 200 according to an embodiment of the inventive concept is illustrated. Three remote production nodes S1, S2, and S3, are interconnected via a network 50 with a receiving node R, which may be a gateway or a processing center through which the data streams M1, M2, and M3 from the respective production nodes S1, S2, and S3 are received and time stamped with a time compensation based on a determined CORR factor, and subsequently retransmitted before arriving in a production studio (not shown in FIG. 3). Alternatively, there is no gateway, but instead the restamping occurs at the studio.

[0062] At each of the production nodes, packets (optionally with the same or correlated specific sequence numbers, respectively) from a source device recording a specific moment in time, are initially time stamped with:

T.sub.stamp1,2,3=t.sub.ref at moment n=t.sub.n.

When received at the receiving node R, each of the packets have an individual delay t.sub.d1, t.sub.d2, t.sub.d3 through the network. The correction factor CORR factor is selected as the minimum delay Δmin of the aggregate set of link delays is determined, e.g. t.sub.d2<t.sub.d1<t.sub.d3=>Δmin=t.sub.d2. To time compensate each of the media streams the respective packets are restamped at the receiver R with a network offset time compensated time:

T.sub.restamp1=t.sub.n+t.sub.d1−Δmin=t.sub.n+t.sub.d1−t.sub.d2

T.sub.restamp2=t.sub.n+t.sub.d2−Δmin=t.sub.n+t.sub.d2−t.sub.d2=t.sub.n

T.sub.restamp3=t.sub.n+t.sub.d3−Δmin=t.sub.n+t.sub.d3−t.sub.d2

Each of the packets that are then forwarded in/retransmitted to the production studio where the time stamp of one of the packets indicates to not experience any delay through the network 50, and where the remaining packets seem to have a smaller link delay through the network. According to the new time stamps, the packets seem to have been “rejuvenated” by removing the minimum delay through the network. This enables the destination equipment to be responsible for absorbing any difference between the min delay and max delay by using in-studio link offset, on a condition that the studio equipment is capable of absorbing the difference between min and max delay. The method optimizes the delay from stadium to studio.

[0063] According to another scenario, and according to an embodiment of the inventive concept, as above at each of the production nodes S1-S3, packets (optionally with the same or correlated specific sequence numbers, respectively) from a source device recording a specific moment in time, are time stamped with:

T.sub.stamp1,2,3=t.sub.ref at moment n=t.sub.n.

When received at the receiving node R, each of the packets have an individual link delay t.sub.d1, t.sub.d2, t.sub.d3 through the network from which a network correction factor selected as e.g. the maximum delay Δmax is determined. To time compensate the media stream each packet is restamped at the receiver R with a network offset time compensated time:

T.sub.restamp1=t.sub.n+Δmax

T.sub.restamp2=t.sub.n+Δmax

T.sub.restamp3=t.sub.n+Δmax

Each of the packets are then directly forwarded in/retransmitted to the production studio and now appear to have experienced the same delay Δmax through the network 50.

[0064] According to an embodiment of the inventive concept, instead of restamping (or in combination with some restamping) the packets, the receiving node R is arranged with buffer capability, and the time compensation of data at R, i.e. the gateway or receiving node, is performed by buffering data received in the data streams and forwarding the buffered data to a receiver or forwards to the processing studio at a time compensated based on the CORR factor, by e.g. adding the network correction factor, e.g. T+Δmax. The network correction factor is preferably selected to an optimum, maximum of within a predetermined margin value.

[0065] Consider now, with reference to FIG. 4 in which a media distribution system 300 according to an embodiment of the inventive concept is illustrated, another scenario and an embodiment of the inventive concept. A remote production node S4 is interconnected via a network 50 with a receiving node R1, which may be a gateway or a processing center through which the data streams M1a, M1b, and M1c from at least one source device is received. The data streams M1 a, M1 b, and M1 c here represent video, audio and ANC-data streams which are time stamped at the source and transmitted to the receiving node R1 for restamping before being further forwarded in/transmitted to a production studio (not shown in FIG. 4). A group of streams such as for example audio can be sent to an intermediate site (not shown in the figure) for processing before being sent to the destination site R1 doing so called distributed production. The intermediate site can also be a data center (private or public cloud) where production processing can be done. As in the previous example, at the production node packets representing video, audio and ANC are represented by an individual time stamp

T.sub.stamp1a,1b,1c=t.sub.ref at moment n=t.sub.n

[0066] When received at the receiving node R1, each of the packets have an individual delay t.sub.d1a, t.sub.d1b, t .sub.d1c through the network from which a network correction factor selected as e.g. the maximum delay Δmax is determined. To time compensate the media stream, each packet is restamped at the receiver R with a network offset time compensated time:

T.sub.restamp1a=t.sub.n+Δmax

T.sub.restamp1b=t.sub.n+Δmax

T.sub.restamp1c=t.sub.n+Δmax

[0067] Each of the packets are then directly retransmitted to the production studio and now appear to have experienced the same delay Δmax through the network 50.

Distributed Production

[0068] According to an embodiment of the inventive concept as illustrated best with reference to FIG. 5, the inventive concept is directed to distributed remote production. In FIG. 5 an exemplifying embodiment is illustrated in which a media distribution system 400 comprises a remote production node S4, which is interconnected via a network 50 with a receiving node R1, which here is the production studio, receiving node R2, which is a cloud processing center, and receiving node R3, which is an audio processing center, through which data streams M1a, M1b, and M1c from at least one source device is received and optionally time compensated, respectively. The data streams M1a, M1b, and M1c (which may be groups of data streams) here represent video, audio and ANC-data streams, respectively, which are time stamped at the source device in production node S1 and transmitted to one of the receiving nodes R1, R2, or R3 for time compensation and/or processing and optionally further transmission to the production studio R1. As in the previous example above, at the production node S4 packets representing video, audio and ANC are represented by an individual time stamp T.sub.stamp1a,1b,1c=t.sub.ref at moment n=t.sub.n. Distributed production can also be done without the remote production concept. In such case, several sites are used to do different parts of a production. It can be that the studio is located in one site, video production is done in another, audio production in a third and subtitling and meta data is processed as a cloud service.

[0069] The video stream M1a is initially optionally compressed at the production node S4 before being transmitted via the network 50 directly to the production studio R1. The total delay for the video stream M1a then adds up to the propagation delay from the production site S4 to the production studio R1 plus time the compression/decompression and optionally further processing time in the production studio R1, while for the audio stream M1b, the total delay comprises its propagation delay from the production site S4 to the audio processing center R3 and to the production studio R1 plus time for audio processing, and for the ANC data stream M1c, the total delay comprises its propagation delay from the production site S4 to the cloud processing center R3 and to the production studio R1 plus time for cloud processing. The CORR factor may be calculated at the production studio R1, in which also time compensation based on a CORR factor determined from the aggregate delays is performed. To coordinate the time stamps between the three sub-groups video, audio and ANC, the longest delay for a sub-group is typically of interest, e.g. the delay for audio. The CORR factor for the incoming may then thus be selected as Δmax, by which all sub-groups are time compensated to coordinate the video-, audio- and ANC data streams associated with the same group, i.e. which belong to production.

[0070] FIG. 7 illustrates the remote media production system 100 as shown in FIG. 1 further comprising a distributed production facility 112 which is operating in the remote. The distributed production studio 112 contains a portion of production equipment 113 needed for processing selected data streams of the audio and video production. The distributed production facility 112 receives a subset M3′ of the data streams from the production nodes and processes them locally before sending them either to the main production facility 110 or directly to a distribution hub. If the data stream is to be processed again or merged with other data streams processed in production facility 110, the stream M3 is treated as other incoming streams in receiving node 110.

Source Time Manipulation

[0071] According to an embodiment of the method, the step of time compensating data comprises at a receiving node determining the network correction factor, CORR-factor, and then sending the CORR factor as feedback to a corresponding production node. The production node is arranged to receive the feedback CORR factor from the receiving node and to time compensate outgoing data streams by time stamping data at at least one source device with a local time stamp compensated by adding the CORR factor selected to be e.g. Δmax [(T+Δmax)]; and immediately transmitting the data. The transmission of the data stream(s) is thus performed without buffering the data streams. FIG. 6 is a schematic illustration of use of an adjusted reference source clock used to generate and send streams from a remote production site to optimize delay at the receiving node to avoid frame buffering. At the local studio an aggregate of individual delays of data streams from a production site is monitored and a network delay correction factor is determined, e.g. the maximum delay Δmax at that particular moment. The network delay correction factor is communicated to the remote site, i.e. the production site. The reference source clock t.sub.ref used to generate and send streams from the remote production site is then adjusted by the network delay correction factor Δmax such that the data streams are stamped with a time t′.sub.Remote=t+Δmax to optimize the perceived delay of the subsequently received stream at the local studio. At a receive time t.sub.Receive the received frame start of received data stamped with t′=12:00:00+Δmax at the local studio thus time wise align with a frame start of local data created at t=12:00:00, the time stamps of the received stream are, which makes it possible to avoid frame buffering. By continuously (or with a predetermined interval) monitoring the aggregate of individual delays of received data streams and determining a selected CORR factor, e.g. Δmax, the communicated CORR factor to the production site can be adjusted to mirror the current status of the network.

Handling Errors

[0072] According to an embodiment of the inventive concept, in addition to time compensating data streams in the receiving node, e.g. a gateway, the method further comprises monitoring time stamps of incoming data streams in the LSET of a group being time compensated with a selected network CORR factor, and comparing subsequent time stamps in the incoming data streams of the group to identify sudden changes in time stamps of the incoming data streams of the group. By determining if the currently selected CORR factor is less than a local offset time, typically about 5 ms, errors in the network may be discovered and if that there is an error in the network a new network CORR factor to compensate for the error is selected. For video streams, to handle the error, skip or repeat of frames may be necessary to compensate for the error.