VARIABLE LINK AGGREGATION

Abstract

A system and method to transmit frames from a first node to a second node over a plurality of radio links comprising a classifier to classify said frames according to one of a plurality of flow and a sequence number within said one of said plurality of flow and adding said flow and sequence number in a header of said classified frame a splitter receiving said classified frames from said classifier and distributing said classified frames on one of said plurality of radio links for transmission to said second node, a joiner receiving said classified frames and reordering them using an indexed sequence queue corresponding to each of said plurality of flows, a timer for waiting for frames missing in the sequence in one of said indexed sequence queue, wherein when said timer expires, if said frame has not arrived it is deemed lost and a forwarder to extract frames from said sequence queue to forward.

Claims

1. A network system comprising: a first network node including: one or more receivers for receiving from a second network node a plurality of frames of a single flow over respective one or more links; a joiner configured to, for each received frame of the single flow, with use of a flow identifier and a sequence identifier stored in the received frame, store in a memory store an indication of receipt of the received frame of the single flow and order the received frame in an index sequence queue for the single flow; and a forwarder configured to, with use of the indications of receipts of the received frames of the single flow, extract from the index sequence queue, the received plurality of frames of the single flow.

2. The network system of claim 1, wherein the forwarder, in its configuration to extract from the index sequence queue the received plurality of frames, is further configured to: determine that a received frame of the single flow is to be removed from the index sequence queue from a presence of an indication of the receipt of a received frame of the single flow in the memory store, at a time subsequent to said determining, extract a received frame from an expected position within the index sequence queue, and remove the indication of the receipt of the received frame of the single flow from the memory store.

3. The network system of claim 1, wherein the forwarder, in its configuration to extract from the index sequence queue the received plurality of frames, is further configured to: determine that a received frame of the single flow is to be removed from the index sequence queue from the presence of an indication of the receipt of a received frame of the single flow in the memory store, determine that the received frame is absent from the expected position within the index sequence queue, wait for a time limit according to a timer for the received frame to be ordered into the expected position within the index sequence queue, after the time limit has expired, determine a frame corresponding to the expected position has been lost, extract a received frame from a next expected position within the index sequence queue, and remove the indication of the receipt of the received frame of the single flow from the memory store.

4. The network system of claim 1, wherein the memory store comprises a flow queue, and the indications of receipts of the received frames of the single flow comprise flow identifiers stored in said flow queue.

5. The network system of claim 4, wherein ordering the received frames in the index sequence queue for the single flow comprises storing the received frames in the indexed sequence queue indexed by the sequence identifier such that entries in the indexed sequence queue corresponding to received frames which have not yet been stored remain empty.

6. The network system of claim 5, wherein the forwarder, in its configuration to extract from the index sequence queue, the received plurality of frames of the single flow, is further configured to: extract the flow identifier from the flow queue, and after extracting the flow identifier from the flow queue, if a next entry of the indexed sequence queue for the single flow is empty, reinsert the flow identifier in the flow queue.

7. The network system of claim 6, wherein the first network node is further configured to wait, using a timer, started upon said reinsertion of the flow identifier in the flow queue, for a received frame to be ordered in the next entry in said indexed sequence queue which is empty, and wherein when said timer expires, if said entry is not occupied, a frame corresponding to said entry is deemed lost, and if said entry is occupied, the received frame occupying said entry is extracted and forwarded and the timer is reset.

8. The network system of claim 1, further comprising: the second network node including: a classifier configured to, for each frame of the plurality of frames, determine that the frame belongs to the single flow and determine a sequence number for the frame within the single flow, insert the flow identifier and the sequence identifier into said frame, and distribute the frame on one of said one or more links for transmission; and one or more transmitters for transmitting said plurality of frames from the second network node to the first node over the respective one or more links.

9. The network system of claim 8, further comprising a splitter configured to distribute each of said plurality of frames on one of said one or more links for transmission, including checking load factors of the one or more links and selecting said one of said one or more links for transmission with use of said load factors.

10. The network system of claim 9, wherein checking the load factors of the one or more links comprises determining the load factors taking into account a size of the frame being distributed.

11. A method of receiving frames in a network system, the method comprising: at a first network node, receiving from a second network node a plurality of frames of a single flow over one or more links; for each received frame of the single flow, with use of a flow identifier and a sequence identifier stored in the received frame, storing in a memory store an indication of receipt of the received frame of the single flow and ordering the received frame in an index sequence queue for the single flow; and extracting from the index sequence queue, the received plurality of frames of the single flow, with use of the indications of receipts of the received frames of the single flow.

12. The method of claim 11, wherein extracting from the index sequence queue the received plurality of frames comprises: determining that a received frame of the single flow is to be removed from the index sequence queue from a presence of an indication of the receipt of a received frame of the single flow in the memory store, at a time subsequent to said determining, extracting a received frame from an expected position within the index sequence queue, and removing the indication of the receipt of the received frame of the single flow from the memory store.

13. The method of claim 11, wherein extracting from the index sequence queue the received plurality of frames further comprises: determining that a received frame of the single flow is to be removed from the index sequence queue from a presence of an indication of receipt of a received frame of the single flow in the memory store, determining that the received frame is absent from an expected position within the index sequence queue, waiting for a time limit according to a timer for the received frame to be ordered into the expected position within the index sequence queue, after the time limit has expired, determining a frame corresponding to the expected position has been lost, extracting a received frame from a next expected position within the index sequence queue, and removing the indication of the receipt of the received frame of the single flow from the memory store.

14. The method of claim 11, wherein the memory store comprises a flow queue, and the indications of receipts of the received frames of the single flow comprise flow identifiers stored in said flow queue.

15. The method of claim 14, wherein ordering the received frames in the index sequence queue for the single flow comprises storing the received frames in the indexed sequence queue indexed by the sequence identifier such that entries in the indexed sequence queue corresponding to received frames which have not yet been stored remain empty.

16. The method of claim 15, wherein extracting from the index sequence queue, the received plurality of frames of the single flow comprises extracting the flow identifier from the flow queue, and after extracting the flow identifier from the flow queue, if a next entry of the indexed sequence queue for the single flow is empty, the flow identifier is reinserted in the flow queue.

17. The method of claim 16, further comprising waiting, using a timer, started upon said reinsertion of the flow identifier in the flow queue, for a received frame to be ordered in the next entry in said indexed sequence queue which is empty, and wherein when said timer expires, if said entry is not occupied, a frame corresponding to said entry is deemed lost, and if said entry is occupied, the received frame occupying said entry is extracted and forwarded and the timer is reset.

18. The method of claim 11, further comprising: at the second network node, for each frame of the plurality of frames, determining that the frame belongs to the single flow and determining a sequence number for the frame within the single flow, inserting the flow identifier and the sequence identifier into said frame, and distributing the frame on one of said one or more links for transmission; and transmitting said plurality of frames from the second network node to the first network node over the one or more links.

19. The method of claim 18, wherein distributing each of said frames on one of said one or more links for transmission, comprises checking load factors of the one or more links and selecting said one of said one or more links for transmission with use of said load factors.

20. The method of claim 19, wherein checking the load factors of the one or more links, comprises determining the load factors taking into account a size of the frame being distributed.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0010] FIG. 1 illustrates a wireless network with variable link aggregation.

[0011] FIG. 2 illustrates an example of frames being classified

[0012] FIG. 3 illustrates an embodiment where frames are split onto multiple links.

[0013] FIG. 4 illustrates an embodiment where frames are combined from the multiple links.

[0014] FIG. 5 illustrates an example of the frame reordering embodiment.

[0015] FIG. 6 illustrates another example of the frame reordering embodiment.

DETAILED DESCRIPTION

[0016] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0017] Like reference numbers and designations in the various drawings indicate like elements.

[0018] FIG. 1 depicts an example embodiment. A first network node 10 receive frames from a network 30. The network 30 may be wired or wireless or a combination thereof. The frames are classified 106 then split across multiple radio transmitters 206a, 206b, 206c for transmission to another node 20. The frames are received at radio receivers 310a, 310b, 310c. They are then joined and sequenced 312 before being forwarded 318 to a network 40.

[0019] In one embodiment, the traffic from a single flow is split across multiple links. This introduces the possibility that individual frames in the single flow arrive out of order at the remote node 20. This embodiment optimally splits traffic from a single flow (or multiple flows) across multiple variable speed wireless links and recombines it efficiently at the remote side so proper flow ordering is maintained while still providing predictable Quality of Service (QoS).

[0020] FIG. 2 depicts an example of how the frames are classified in preparation for splitting on several links. Frames 102 ready to be transmitted on a radio link arrive at a classifier 106. The size of the box 102 exemplifies a relative size of the frames 102. The number in the box is the arrival sequence number 104 of the frame. The classifier processes the headers to determine the flow identifier of each frame (F) a sequence identifier (S) is also added to each frame 102. In this example, there are four flows 108a, 108b, 108c, 108d. And each flow has one or more frames arriving in a sequence. In this example, frame #1,#3 and #8 belong to flow A (108a) The flow identifier (F) a sequence identifier (S) is added to each header 110. In this example, “A/1” indicates the first frame of flow A, “A/2” the second frame in flow A.

[0021] FIG. 3 depicts classified frames arriving at a splitter 202. The splitter 202 checks the load factor 204a, 204b, 204c of each radio channel that would result if the current frame was sent to that channel. Then picks the least loaded channel to forward the frame. Calculating the load factor by including the current frame is optional but can improve latency because smaller frames can be sent on radio links 210a, 210b, 210c with fastest transmission bit rate (all else being equal). The link monitor 206a, 206b, 206c monitors the radio 210a, 210b, 210c and update the load meter 204a, 204b, 204c parameters dynamically. If the link goes down, the transmit bit rates goes to zero and the load factor is set to infinity.

[0022] FIG. 4 depicts the receive and reassembly side implemented on the receiving radio 310a, 310b, 310c. A joiner function 312 reads frames from radios on a first come first serve basis. It extracts the flow identifier (F) a sequence identifier (S) from each header, then directs the frames to the appropriate sequence queue 316a, 316b, 316c, 316d based on the flow identifier (F) and slots into each queue based on the sequence identifier (S). A sequence queue is a slotted queue that holds frames indexed by sequence identifier (S). The queues handle out of order and dropped frames using timers.

[0023] In a preferred embodiment, the flow identifier is optionally placed in a Flow Queue 314. A forwarder 318 reads the flow identifier at the top of the flow queue 314 and uses it to access the appropriate sequence queue to read from. If there is no frame at the head (dropped or out of order) then the flow identifier is requeued in the flow queue and a timer is started for this flow queue. If a frame is available at the head of the sequence queue, the forwarder reads the frame and forwards it, and if the timer was previously started, it is reset. If there is no frame at the head and the timer has expired, then the forwarder assumes this frame is lost and advances to the next frame in the queue.

[0024] The frames exit the forwarder 320 in the right order with respect to their specific flow but can be reordered with respect to other flows and the original frame arrival 322 which does not create an issue.

[0025] FIG. 5 illustrates an example of the frame arrival in sequence queue A and the process to reorder the frames. In this example the frame for flow A 316a with the first sequence number arrives first 510 and the joiner places the frame in the sequence queue and puts A into the flow queue 314. Then the frame with the 3.sup.rd sequence number arrives next 512, before the 2.sup.nd. The joiner places the third frame in its location in the sequence queue and puts another A in the flow queue 314. The space for the second frame remains empty 514. When the forwarder reads A from the flow queue 314, it reads the first frame 510 and transmits it. The next time the forwarder reads A in the flow queue 314, it sees an empty (“NULL”) 514 entry in the sequence queue 316b, a timer 520 is started and the forwarder reinserts A back into the flow queue 314. When the forwarder reads A again from the flow queue 314, if the timer expired, packet #2 of the flow A is deemed lost and the forwarder reads and transmits the next packet 512.

[0026] FIG. 6 illustrates another example of the frame arrival a sequence queue 316a and the process to reorder the frames. In this example the frame for flow A with the first sequence number arrives first 510 and the joiner places the frame in the sequence queue and puts A in the flow queue 314. Then the frame with the 3.sup.rd sequence number arrives next 512, before the 2.sup.nd. The third frame is placed in its location in the sequence queue and the joiner puts another A in the flow queue 314 and the space for the second frame remains empty 514.

[0027] When the forwarder reads A from the flow queue 314, it reads the first frame 510 and transmits it. The next time the forwarder reads A from the flow queue 314, it sees an empty slot (“NULL”) 514 entry in the sequence queue 316b, a timer 520 is started and the forwarder reinserts A back into the flow queue 314. When the forwarder reads A again from the flow queue 314, packet #2 518 has arrived and the forwarder reads and transmits it, then resets the timer.

[0028] There are a limited number of sequence numbers and the system can use a modulo function to wrap around the number. For example, using 8 bits, there can be 256 sequence numbers (0-255).

[0029] The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention and method of use to the precise forms disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments described were chosen and described in order to best explain the principles of the invention and its practical application, and to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is understood that various omissions or substitutions of equivalents are contemplated as circumstance may suggest or render expedient but is intended to cover the application or implementation without departing from the spirit or scope of the claims of the present invention.