VIDEO DATA BURST CONTROL FOR REMOTE TOWERS
20220030225 · 2022-01-27
Inventors
Cpc classification
H04N19/159
ELECTRICITY
H04N19/114
ELECTRICITY
H04N21/26275
ELECTRICITY
H04N19/107
ELECTRICITY
H04N21/23655
ELECTRICITY
International classification
H04N19/107
ELECTRICITY
H04N19/114
ELECTRICITY
H04N19/159
ELECTRICITY
Abstract
The present invention relates to a method for managing data output from multiple cameras of a remote tower via a common transmission media to a central entity, non-transitory computer-readable storage medium, and a corresponding remote tower system. The method comprises determining a global GOP sequence for the multiple cameras, determining a first distribution of I-frames within the global GOP sequence and forming a second distribution of I-frames within that global GOP sequence by moving the I-frame of at least one camera. The proposed method allows for transmitting data from a remote virtual tower to a control centre with reduced data peaks whereby the data can be transmitted without additional buffering and the end-to-end transmission delays can be kept at low level.
Claims
1-15. (canceled)
16. A method for managing data output from multiple cameras of a remote tower via a common transmission media to a central entity, wherein each camera is configured to periodically output an Intra frame, in a Group of Pictures, GOP, sequence, the method comprising: determining a global GOP sequence for the multiple cameras, said global GOP sequence comprising the Intra frame of each camera of the multiple cameras; determining a first distribution of Intra frames within the global GOP sequence; and forming a second distribution of Intra frames within the global GOP sequence by moving the Intra frame of at least one of said multiple cameras within the global GOP sequence so that the second distribution is different from the first distribution.
17. The method according to claim 17, wherein: the method further comprises identifying, in the first distribution of Intra frames, at least one time slot comprising at least two Intra frames, said time slot being of a limited length shorter than the global GOP sequence; and the step of forming the second distribution comprises moving at least one Intra frame of the at least two Intra frames from said at least one time slot to a different time slot within the global GOP sequence.
18. The method according to claim 16, wherein: the step of determining a first distribution of Intra frames within the global GOP sequence comprises: dividing the global GOP sequence into a plurality of subsections; identifying at least one high-density subsection, wherein the high-density subsection comprises a number of Intra frames above a first predefined threshold; and identifying at least one low-density subsection, wherein the low-density subsection comprises a number of Intra frames below a second predefined threshold; and the step of forming the second distribution within the global GOP sequence comprises moving an Intra frame from each identified high-density subsection to a corresponding low-density subsection.
19. The method according to claim 18, wherein the first predefined threshold and the second predefined threshold is equal to one.
20. The method according to claim 18, wherein: the global GOP sequence comprises N Intra frames, wherein the step of dividing the global GOP sequence into a plurality of subsections comprises dividing the global GOP sequence into N subsections; and the step of forming the second distribution within the global GOP sequence comprises moving an Intra frame from each identified high-density subsection to a corresponding low-density subsection such that each subsection out of the N subsections comprises one Intra frame.
21. The method according to claim 20, wherein the step of dividing the global GOP sequence into N subsections comprises: forming a group of subsections comprising a series of the N subsections; and aligning the group of subsections with said global GOP sequence based on a predefined metric.
22. The method according to claim 18, wherein the step of forming the second distribution within the global GOP sequence further comprises changing a status of each low-density subsection to a normal-density subsection when the Intra frame is moved thereto.
23. The method according to claim 18, wherein the subsections are of equal length.
24. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a remote tower system, the one or more programs comprising instructions for performing the method according to claim 16.
25. A remote tower system for air traffic control comprising: a plurality of cameras arranged to transmit data output via a common transmission media to a central entity, each camera being configured to periodically output an Intra frame in a Group of Pictures, GOP, sequence; and a controller arranged to monitor and control the data output of each camera onto the common transmission media, wherein the controller is configured to: determine a global GOP sequence for the plurality of cameras, the global GOP sequence comprising the Intra frame of each camera of the plurality of cameras; determine a first distribution of Intra frames within the global GOP sequence; and form a second distribution of Intra frames within the global GOP sequence by moving the Intra frame of at least one of the plurality of cameras within the global GOP sequence so that the second distribution is different from the first distribution.
26. The remote tower system according to claim 25, wherein the controller is further configured to: identify, in the first distribution of Intra frames, at least one time slot comprising at least two Intra frames, said time slot being of a limited length shorter than the global GOP sequence; and form the second distribution of Intra frames within the global GOP sequence by moving at least one Intra frame of the at least two Intra frames from said at least one time slot to a different time slot within the global GOP sequence.
27. The remote tower system according to claim 25, wherein the controller is further configured to: arrange the global GOP sequence into a plurality of subsections; identify at least one high-density subsection, wherein the high-density subsection comprises a number of Intra frames above a first predefined threshold; identify at least one low-density subsection, wherein the low-density subsection comprises a number of Intra frames below a second predefined threshold; and form the second distribution of Intra frames within the global GOP sequence by moving an Intra frame from each identified high-density subsection to a corresponding low-density subsection.
28. The remote tower system according to claim 27, wherein the global GOP sequence comprises N Intra frames, and wherein the controller is configured to: arrange the global GOP sequence into N subsections; and form the second distribution of Intra frames within the global GOP sequence by moving an Intra frame from each identified high-density subsection to a corresponding low-density subsection such that each subsection out of the N subsections comprises one Intra frame.
29. The remote tower system according to claim 28, wherein the controller is configured to arrange the global GOP sequence into N subsections by forming a group of subsections comprising the N subsections, and aligning the group of subsections with said global GOP sequence based on a predefined metric.
30. The remote tower system according to claim 27, wherein the controller is further configured to change a status of each low-density subsection to a normal-density subsection when the Intra frame is moved thereto.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0050] For exemplifying purposes, the invention will be described in closer detail in the following with reference to embodiments thereof illustrated in the attached drawings, wherein:
[0051]
[0052]
[0053]
[0054]
[0055]
[0056]
DETAILED DESCRIPTION
[0057] In the following detailed description, some embodiments of the present invention will be described. However, it is to be understood that features of the different embodiments are exchangeable between the embodiments and may be combined in different ways, unless anything else is specifically indicated. Even though in the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well known constructions or functions are not described in detail, so as not to obscure the present invention.
[0058]
[0059] The remote tower 4 may for example comprise more than ten cameras (such as e.g. twelve or fourteen cameras) arranged in a camera house. The camera house is arranged to protect the cameras from weather (rain, snow, hail, moisture, etc.), high and low temperatures, sunlight, insects, birds, etc. The cameras 9 may be any type of suitable optical sensors such as e.g. high definition video cameras, infrared cameras, night vision cameras, etc. Moreover, the remote tower may also be realized as a plurality of cameras or groups of cameras spatially distributed at different locations (not shown) overlooking the airport, instead of being gathered in a single tower-like structure as illustrated in the drawing.
[0060] As the skilled artisan realizes the remote tower 4 further may further be provided with radar systems, microphones, and any other suitable arrangements used in general ATC applications. These aspects are however considered obvious for the skilled reader and will for the sake of brevity and conciseness not be further discussed in any detail.
[0061] Moving on, the plurality of cameras 9 are arranged to transmit a data output (i.e. an image feed or video feed) via a common transmission media 2 (e.g. wide area network) to a central entity 3 (e.g. control centre). The term common with respect to the common transmission media is to be understood as “shared by” and not as “ordinary” or “conventional”. Moreover, the common transmission media may be realised by a wired connection, wireless connection, or a combination thereof. The data output, i.e. the video output is digitally encoded by a method that uses I-frames together with Predicted Frames (P-frames) and possibly Bi-directional Frames (B-frames). Accordingly, each camera 9 is configured to generate an image output (video output) comprising an I-frame (Intra frame) in a Group of Pictures (GOP) sequence.
[0062] I-frames are larger than the other frames (P-frames/B-frames) and can be said to form peaks in the output bandwidth from the cameras 9. The I-frames can be said to contain the full image and do therefore not require any additional information to reconstruct them. Typically, encoders use GOP structures that cause each I-frame to be a “clean random access point,” such that decoding can start cleanly on an I-frame.
[0063] In these compression standards (video coding designs), the full frame (I-frame) is stored only at specific intervals of e.g. one per second, and encodes the rest of the frames (P-frames and B-frames) only with the differences caused by motion in the video. The time interval for I-frames varies (e.g. multiple times per second, or once every second, once every 10 seconds, etc.). The more I-frames that are to be transmitted, the larger the video stream will be which requires more bandwidth capacity. However, the more I-frames the video stream has, the more editable it is.
[0064] An example of data output from a camera 9 can be seen in
[0065] In more detail, a GOP may include the following picture types: [0066] I-frame (intra/inter coded picture). An I-frame is a picture that is coded independently of all other pictures. Each GOP begins (in decoding order) with this type of picture. [0067] P-frame (predictive coded picture). The P-frame contains motion-compensated difference information relative to previously decoded pictures. [0068] B-frame (bi-predictive coded picture). A B-frame also contains motion-compensated difference information relative to previously decoded pictures.
[0069] Further, the remote tower system 1 has a controller 10 (may also be referred to as a control unit), here as a part of a remote server/controller, such as e.g. a cloud-based system 5, arranged to monitor and control the data output of each camera 9 onto the common transmission media 2. More specifically, the controller 10 is arranged to monitor the data output from each camera in real-time, and is capable of moving the GOP sequence for each individual camera by e.g. an external command.
[0070] The controller 10 is configured to determine a global GOP sequence for the plurality of cameras 9, where the global GOP sequence comprises the I-frame of each camera of the plurality of cameras 9. Thereafter, the controller 10 analyses the global GOP sequence and determines or observes a first distribution of I-frames within the GOP. An example of this can be seen in
[0071] Further, the controller 10 is configured to form a second distribution of I-frames within the global GOP sequence by moving the I-frame of at least one of the plurality of cameras 9 within the global GOP sequence so that the second distribution is different from the first distribution. For example, the controller 10 may be configured to form a second distribution of I-frames within the global GOP sequence by moving the I-frame of at least one of the plurality of cameras 9 within the global GOP sequence so that the I-frames are more evenly distributed within the global GOP sequence as compared to the first distribution.
[0072] The step of moving an I-frame may for example include shortening a single GOP sequence for a specific camera so that the future I-frames are being generated and transmitted at their designated positions within the global GOP sequence. The shortening can be done by e.g. requesting a new I-frame at a specific point in time within the global GOP sequence, which will consequently move the start of individual GOP sequence to that point in time. However, the step of moving an I-frame may alternatively be done by extending a single GOP sequence for a specific camera so that the future I-frames are being generated and transmitted at their designated positions within the global GOP sequence.
[0073] An example of a resulting second distribution is illustrated in
[0074]
[0075] Further, a first distribution of I-frames 50a within the global GOP sequence 51 is determined 102. This may for example be construed as forming a timeline 52 and recording, on the timeline, when each camera emits an I-frame 50a. Each I-frame 50a is accordingly marked with a representation 50b on the timeline 52. The start 53 of the global GOP sequence 51 and the end of the global GOP sequence 54 accordingly serve as a start and stop of the “recording”. Next, the global GOP sequence 51 is divided 104 into a plurality of subsections 55. Preferably, the global GOP sequence 51 is divided into as many subsections 55 as there are I-frames 50a within the global GOP 51. In the illustrated example, the global GOP sequence 51 comprises the data output from four cameras, and accordingly includes four I-frames 50a resulting in four subsections 55. Alternatively, the method 100 may comprise a step of identifying subsections 55 in the formed global GOP sequence timeline 52.
[0076] The subsections 55 are then arranged in a group of subsections 55. The group includes the individual subsections arranged in a sequential manner. The total length (in time) of the group of subsections 55 is equal to the length (in time) of the global GOP sequence 51.
[0077] Moving on, high-density subsections 55a and low-density subsections 55b are identified 106. A high-density subsection 55a can be construed as a subsection having a relatively high-density of I-frames, e.g. two or more I-frames 50a. In more detail, a high-density subsection 55a can be understood as a subsection of the global GOP sequence that is more likely to exceed available bandwidth on the common transmission media than other subsections (low and normal-density subsections). A low-density subsection 55b can be construed as subsection having a relatively low-density of I-frames 50a, e.g. zero I-frames. In more detail, a low-density subsection 55b can be understood as a subsection in the global GOP sequence that into which one can insert additional I-frames without exceeding the bandwidth of the common transmission media. The amount of data that has to be transmitted per second from the plurality of cameras is referred to as bandwidth. It is generally measured in bit/s, but may also be measured in byte/s which equals to ⅛.sup.th of the corresponding bitrate.
[0078] Moreover, the method 100 may further comprise identifying or classifying the remaining subsections as normal subsections 55c. Accordingly, the step of classifying 106 subsections 55 in the group of subsections 55 may for example be such that: [0079] If the subsection contains zero I-frames, that section is classified/identified as a low-density subsection. [0080] If the subsection contains two or more I-frames, that section is classified/identified as a high-density subsection. [0081] If the subsection contains one and only one I-frame, that section is classified/identified as a normal-density subsection.
[0082] As an option, the method may include a step of aligning 105 the group of subsections 55 by some predefined metric before identifying/classifying the subsections as high, low and normal-density. More specifically, the alignment 105 or adjustment may be understood optimizing the placement of the group of subsections 55 on the formed timeline 52 representation of the global GOP sequence 51. The predefined metric may for example be to minimize the sum of absolute differences between the time stamp of an I-frame 50a and the closest central point of a subsection 55. In other words, the step of aligning 105 the group of subsections may comprise adjusting the position in time of the group of subsections 55 such that a sum of an absolute difference between a time stamp of an I-frame 50a and a closest central point of a subsection is below a first predefined threshold.
[0083] Another example of a predefined metric is to minimize the sum of the squared differences between the time stamp of an I-frame 50a and the closest central point of a section 55. In other words, the step of aligning 105 the group of subsections may comprise adjusting the position in time of the group of subsections 55 such that a sum of a squared difference between a time stamp of an I-frame 50a and a closest central point of a section 55 is below a second predefined threshold.
[0084] Further, the method 100 includes a step of forming 103 a second distribution of I-frames 50 within the global GOP sequence 51 by moving 107 the I-frame 50a of one or more cameras within the global GOP sequence 51 such that the second distribution is different from the first distribution. In more detail, one of the I-frames 50a in the high-density subsection 55a is moved to the low-density subsection 55b. More specifically, the I-frame 50a is moved to a different point in time within the interval defined by the low-density subsection 55b. Preferably, the I-frame 50a is moved to a center point of the interval defined by the low-density subsection 55b. Once, all of the relevant Intra frames 50a are moved and the second distribution is formed, as indicated by the time-line 56 showing representations (marked with an x) of the position of each I-frame 50a in the second distribution of I-frames 50a within the global GOP sequence 51. The method may further comprise a step of changing 108 the status of the subsections from a high/low-density subsection to a normal subsection 55c.
[0085] Thus, on a general level, the forming 103 of the second distribution may include populating a list {A} with I-frames 50a that are out of order by taking I-frames 50a from high-density subsections 55a until there are no more high-density subsections left. For each selected I-frame 50a the method 100 may also include making a note of where it shall be moved within the global GOP sequence (i.e. to a free low-density subsection 55b).
[0086]
[0087] The system 1 further has a controller 10 connected to each camera 9. The controller 10 is arranged to monitor 8 and control the data output (in real-time) of each camera 9 onto the common transmission media 2. Even though only three cameras 9 are illustrated in
[0088] It should be noted that the controller 10 may for example be manifested as a general-purpose processor, an application specific processor, a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, a field programmable gate array (FPGA), etc. The controller 10 may for example be in the form of a circuit having a processor 11 such as e.g. a microprocessor, microcontroller, programmable digital signal processor or another programmable device. The controller 10 may also, or instead, include an application-specific integrated circuit (ASIC), a programmable gate array or programmable array logic, a programmable logic device, or a digital signal processor. Where the controller 10 includes a programmable device such as the microprocessor, microcontroller or programmable digital signal processor mentioned above, the processor 11 or an associated memory 12 may further include computer executable code that controls operation of the programmable device.
[0089] It should be understood that the controller 10 may comprise a digital signal processor arranged and be configured for digital communication with an off-site server or cloud based server. Analogously, if the controller 10 is part of a cloud based system (as depicted in
[0090] Further, it should be understood that parts of the described solution may be implemented either in the controller 10, in a system located external the controller 10, or in a combination of internal and external the controller 10; for instance in a server in communication with the controller 10, a so called cloud solution. For instance, communication signal may be sent to an external system and that external system performs the steps to determine how to move the I-frames so to form the second distribution and send back information indicating the moving process and other relevant parameters used in forming the second distribution.
[0091] The processor 11 (of the controller 10) may be or include any number of hardware components for conducting data or signal processing or for executing computer code stored in memory 12. Accordingly, controller 10 may have an associated memory 12, and the memory 12 may be one or more devices for storing data and/or computer code for completing or facilitating the various methods described in the present description. The memory 12 may include volatile memory or non-volatile memory. The memory 12 may include database components, object code components, script components, or any other type of information structure for supporting the various activities of the present description. According to an exemplary embodiment, any distributed or local memory device may be utilized with the systems and methods of this description. According to an exemplary embodiment the memory 12 is communicably connected to the processor 11 (e.g., via a circuit or any other wired, wireless, or network connection) and includes computer code for executing one or more processes described herein.
[0092] Moreover, depending on functionality provided in the controller 10 one or more communication interfaces 13, 14 and/or one or more antenna interfaces (not shown) may be provided and furthermore, also one or more sensor interfaces (not shown) may be provided for acquiring data from sensors associated with the system.
[0093] Moving on, the controller 10 is configured to determine a global GOP sequence for the plurality of cameras 9. The global GOP sequence comprises one I-frame of each camera in the plurality of cameras. The global GOP sequence is a continuous or discrete time series, thus it can be construed as the smallest interval in time of the data output on the common transmission media 2 in which one can fit the I-frame from each camera 9.
[0094] Further, the controller is configured to determine a first distribution of I-frames in the global GOP sequence. This step may also be construed as detecting or observing a first distribution (i.e. unordered distribution) of I-frames within the determined global GOP sequence. Next, the control unit 10 is configured to form a second distribution (i.e. ordered distribution) of I-frames within the global GOP sequence by moving the I-frame of at least one camera 9 within the global GOP sequence so that the second distribution is different form the first distribution.
[0095] The controller 10 may be configured to perform any one of the method steps discussed in the foregoing in reference to
[0096] Preferably, the controller 10 is configured to ensure that no high-density subsections are formed in the second distribution.
[0097] In summary, the inventive method may be construed as, on a periodic basis, analysing a current distribution of I-frames within the global GOP, identifying a list {A} containing I-frames that are out of order, designating a new position for each I-frame in the list {A} within the global GOP sequence, and moving one or more I-frames from the list {A} to their new designated positions within the global GOP sequence. The event that invokes periodic reordering can be based on a periodic timer event, on the global GOP sequence, or an external command (e.g. the system controller noticing an increasing risk of congestion in the data traffic).
[0098] The present disclosure contemplates methods, devices and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor.
[0099] By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions. As already exemplified, some parts or all of the functions may be realized as a “cloud-based” solution.
[0100] Although the figures may show a specific order of method steps, the order of the steps may differ from what is depicted. In addition, two or more steps may be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps. Additionally, even though the disclosure has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art.
[0101] It should be noted that the word “comprising” does not exclude the presence of other elements or steps than those listed and the words “a” or “an” preceding an element do not exclude the presence of a plurality of such elements. It should further be noted that any reference signs do not limit the scope of the claims, that the invention may be at least in part implemented by means of both hardware and software, and that several “means” or “units” may be represented by the same item of hardware.
[0102] The above mentioned and described embodiments are only given as examples and should not be limiting to the present invention. Other solutions, uses, objectives, and functions within the scope of the invention as claimed in the below described patent embodiments should be apparent for the person skilled in the art.