SYSTEMS, APPARATUS AND METHODS FOR RENDERING DIGITAL CONTENT

Abstract

Instructions are received at, or transferred to, a client device that includes a display, to cause the display to render a video relating to a sporting event and also render online gaming information. The instructions cause the client device to: receive, on a first communication channel, first digital content corresponding to a replay video of a recording relating to the sporting event; render the replay video based on the first digital content; receive, on a second communication channel different from the first communication channel, second digital content corresponding to the online gaming information, wherein at least some of the online gaming information relates to at least one bet; and render the online gaming information based on the second digital content, wherein the rendered online gaming information includes at least one interactive display widget that renders display content relating to the at least one bet.

Claims

1. A method, comprising: A) receiving first instructions at a first client device (200) that includes at least one first display (250-1) to cause the at least one first display of the first client device to render a first video (252-1) relating to a first sporting event and also render online gaming information (254-1B), wherein the first instructions received in A) cause the first client device to: receive, on a first communication channel, first digital content corresponding to the first video relating to the first sporting event, wherein the first video is a first replay video of a recording relating to the first sporting event; render, on the at least one first display of the first client device, the first replay video based on the first digital content received on the first communication channel; receive, on a second communication channel different from the first communication channel, second digital content corresponding to the online gaming information, wherein: at least some of the online gaming information that corresponds to the second digital content relates to at least one bet; the online gaming information relates to the first sporting event or a second sporting event that is related to the first sporting event; and the at least one bet relates to the first sporting event or the second sporting event; and render, on the at least one first display of the first client device, the online gaming information based on the second digital content received on the second communication channel, wherein the rendered online gaming information includes at least one interactive display widget that renders display content relating to the at least one bet.

2. The method of claim 1, wherein at least one team or at least one participant are featured in both of the first sporting event and the second sporting event.

3. The method of claim 1, wherein at least one league or at least one organization is featured in both of the first sporting event and the second sporting event.

4. The method of claim 1, wherein both of the first sporting event and the second sporting event relate to a same sport.

5. The method of claim 1, wherein the rendered content in the at least one interactive display widget relating to the at least one bet includes at least one of: general betting information; or specific betting information relating to at least one occurrence, at least one milestone, or at least one target statistic.

6. The method of claim 1, wherein: rendering, on the at least one first display of the first client device, the first replay video comprises rendering the first replay video on a first portion of the at least one first display; and rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

7. The method of claim 1, wherein: the second digital content received on the second communication channel corresponds to the online gaming information and score information; the method further comprises rendering, on the at least one first display of the first client device, the score information and the online gaming information based on the second digital content received on the second communication channel; the rendered online gaming information includes the at least one interactive display widget that renders the content relating to the at least one bet; rendering, on the at least one first display of the first client device, the first replay video comprises rendering the first replay video on a first portion of the at least one first display; and the score information is rendered as an overlay on the first portion of the at least one first display.

8. The method of claim 7, wherein the score information is rendered on the at least one first display as at least one interactive overlay.

9. The method of claim 7, wherein: rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

10. The method of claim 1, wherein in A), the first instructions received by the first client device include a first address for a first media source, such that the first client device uses the first address to request and receive from the first media source the first digital content via the first video communication channel.

11. The method of claim 10, wherein in A), the first instructions received by the first client device include a first event identifier (first Event ID) that corresponds to a source of the second digital content corresponding to the online gaming information.

12. The method of claim 11, further comprising: B) transmitting from the first client device a first URL including the first event identifier; and C) communicating over the second communication channel between the source of the second digital content and the first client device as a persistent connection based at least in part on B).

13. The method of claim 1, wherein in A), the first instructions received by the first client device cause the at least one first display of the first client device to render at least some of the online gaming information as interactive content.

14. The method of claim 13, wherein in A), the first instructions received by the first client device cause the first client device, when a user of the first client device interacts with the interactive content, to: launch further graphics or animations; receive additional information about the first sporting event; and/or navigate to an Internet location.

15. A method, comprising: A) transmitting first instructions to a first client device (200) that includes at least one first display (250-1) to cause the at least one first display of the first client device to render a first video (252-1) relating to a first sporting event and also render online gaming information (254-1B), wherein the first instructions received in A) cause the first client device to: receive, on a first communication channel, first digital content corresponding to the first video relating to the first sporting event, wherein the first video is a first replay video of a recording relating to the first sporting event; render, on the at least one first display of the first client device, the first replay video based on the first digital content received on the first communication channel; receive, on a second communication channel different from the first communication channel, second digital content corresponding to the online gaming information, wherein: at least some of the online gaming information that corresponds to the second digital content relates to at least one bet; the online gaming information relates to the first sporting event or a second sporting event that is related to the first sporting event; and the at least one bet relates to the first sporting event or the second sporting event; and render, on the at least one first display of the first client device, the online gaming information based on the second digital content received on the second communication channel, wherein the rendered online gaming information includes at least one interactive display widget that renders display content relating to the at least one bet.

16. The method of claim 15, wherein at least one team or at least one participant are featured in both of the first sporting event and the second sporting event.

17. The method of claim 15, wherein at least one league or at least one organization is featured in both of the first sporting event and the second sporting event.

18. The method of claim 15, wherein both of the first sporting event and the second sporting event relate to a same sport.

19. The method of claim 15, wherein the rendered content in the at least one interactive display widget relating to the at least one bet includes at least one of: general betting information; or specific betting information relating to at least one occurrence, at least one milestone, or at least one target statistic.

20. The method of claim 15, wherein: rendering, on the at least one first display of the first client device, the first replay video comprises rendering the first replay video on a first portion of the at least one first display; and rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

21. The method of claim 15, wherein: the second digital content received on the second communication channel corresponds to the online gaming information and score information; the first instructions transmitted in A) cause the first client device to render, on the at least one first display of the first client device, the score information and the online gaming information based on the second digital content received on the second communication channel; the rendered online gaming information includes the at least one interactive display widget that renders the content relating to the at least one bet; rendering, on the at least one first display of the first client device, the first replay video comprises rendering the first replay video on a first portion of the at least one first display; and the score information is rendered as an overlay on the first portion of the at least one first display.

22. The method of claim 21, wherein the score information is rendered on the at least one first display as at least one interactive overlay.

23. The method of claim 21, wherein: rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

24. The method of claim 15, wherein in A), the first instructions transmitted to the first client device include a first address for a first media source, such that the first client device uses the first address to request and receive from the first media source the first digital content via the first video communication channel.

25. The method of claim 24, wherein in A), the first instructions transmitted to the first client device include a first event identifier (first Event ID) that corresponds to a source of the second digital content corresponding to the online gaming information.

26. The method of claim 25, further comprising: B) receiving from the first client device a first URL including the first event identifier; and C) establishing the second communication channel between the source of the second digital content and the first client device as a persistent connection based at least in part on B).

27. The method of claim 15, wherein in A), the first instructions transmitted to the first client device cause the at least one first display of the first client device to render at least some of the online gaming information as interactive content.

28. The method of claim 27, wherein in A), the first instructions transmitted to the first client device cause the first client device, when a user of the first client device interacts with the interactive content, to: launch further graphics or animations; receive additional information about the first sporting event; and/or navigate to an Internet location.

29. A method, comprising: A) receiving first instructions at a first client device (200) that includes at least one first display (250-1) to cause the at least one first display of the first client device to render a first video (252-1) relating to a first sporting event and render online gaming information (254-1B) relating to the first sporting event, wherein the first instructions received in A) cause the first client device to: receive, on a first communication channel, first digital content corresponding to the first video relating to the first sporting event; render, on the at least one first display of the first client device, the first video relating to the first sporting event based on the first digital content received on the first communication channel; receive, on a second communication channel different from the first communication channel, second digital content corresponding to the online gaming information, wherein at least some of the online gaming information that corresponds to the second digital content relates to at least one bet; and render, on the at least one first display of the first client device, the online gaming information based on the second digital content received on the second communication channel, wherein the rendered online gaming information includes at least one display widget that renders display content relating to the at least one bet.

30. The method of claim 29, wherein the rendered content in the at least one display widget relating to the at least one bet includes at least one of: general betting information; or specific betting information relating to at least one occurrence, at least one milestone, or at least one target statistic.

31. The method of claim 30, wherein the at least one display widget relating to the at least one bet is rendered on the at least one first display of the first client device as at least one interactive display widget.

32. The method of claim 31, wherein: rendering, on the at least one first display of the first client device, the first video relating to the first sporting event comprises rendering the first video on a first portion of the at least one first display; and rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

33. The method of claim 31, wherein: the second digital content received on the second communication channel corresponds to the online gaming information and score information; the method further comprises rendering, on the at least one first display of the first client device, the score information and the online gaming information based on the second digital content received on the second communication channel; the rendered online gaming information includes at least one display widget that renders the content relating to the at least one bet; rendering, on the at least one first display of the first client device, the first video relating to the first sporting event comprises rendering the first video on a first portion of the at least one first display; and the score information is rendered as an overlay on the first portion of the at least one first display.

34. The method of claim 33, wherein the score information is rendered on the at least one first display as at least one interactive overlay.

35. The method of claim 33, wherein: rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

36. The method of claim 29, wherein the first video is a first replay video of a recording relating to the first sporting event.

37. The method of claim 36, wherein: the online gaming information relates to the first sporting event or a second sporting event that is related to the first sporting event; and the at least one bet relates to the first sporting event or the second sporting event.

38. The method of claim 37, wherein at least one team or at least one participant are featured in both of the first sporting event and the second sporting event.

39. The method of claim 37, wherein at least one league or at least one organization is featured in both of the first sporting event and the second sporting event.

40. The method of claim 37, wherein both of the first sporting event and the second sporting event relate to a same sport.

41. The method of claim 29, wherein in A), the first instructions received by the first client device include a first address for a first media source, such that the first client device uses the first address to request and receive from the first media source the first digital content via the first video communication channel.

42. The method of claim 41, wherein in A), the first instructions received by the first client device include a first event identifier (first Event ID) that corresponds to a source of the second digital content corresponding to the online gaming information.

43. The method of claim 42, further comprising: B) transmitting from the first client device a first URL including the first event identifier; and C) communicating over the second communication channel between the source of the second digital content and the first client device as a persistent connection based at least in part on B).

44. The method of claim 29, wherein in A), the first instructions received by the first client device cause the at least one first display of the first client device to render at least some of the online gaming information as interactive content.

45. The method of claim 44, wherein in A), the first instructions received by the first client device cause the first client device, when a user of the first client device interacts with the interactive content, to: launch further graphics or animations; receive additional information about the first sporting event; and/or navigate to an Internet location.

46. A method, comprising: A) transmitting first instructions to a first client device (200) that includes at least one first display (250-1) to cause the at least one first display of the first client device to render a first video (252-1) relating to a first sporting event and render online gaming information (254-1B) relating to the first sporting event, wherein the first instructions transmitted in A) cause the first client device to: receive, on a first communication channel, first digital content corresponding to the first video relating to the first sporting event; render, on the at least one first display of the first client device, the first video relating to the first sporting event based on the first digital content received on the first communication channel; receive, on a second communication channel different from the first communication channel, second digital content corresponding to the online gaming information, wherein at least some of the online gaming information that corresponds to the second digital content relates to at least one bet; and render, on the at least one first display of the first client device, the online gaming information based on the second digital content received on the second communication channel, wherein the rendered online gaming information includes at least one display widget that renders content relating to the at least one bet.

47. The method of claim 46, wherein the rendered content in the at least one display widget relating to the at least one bet includes at least one of: general betting information; or specific betting information relating to at least one occurrence, at least one milestone, or at least one target statistic.

48. The method of claim 46, wherein the at least one display widget relating to the at least one bet is rendered on the at least one first display of the first client device as at least one interactive display widget.

49. The method of claim 46, wherein: rendering, on the at least one first display of the first client device, the first video relating to the first sporting event comprises rendering the first video on a first portion of the at least one first display; and rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

50. The method of claim 46, wherein: the second digital content received on the second communication channel corresponds to the online gaming information and score information; the first instructions transmitted in A) cause the first client device to render, on the at least one first display of the first client device, the score information and the online gaming information based on the second digital content received on the second communication channel; the rendered online gaming information includes at least one display widget that renders the content relating to the at least one bet; rendering, on the at least one first display of the first client device, the first video relating to the first sporting event comprises rendering the first video on a first portion of the at least one first display; and the score information is rendered as an overlay on the first portion of the at least one first display.

51. The method of claim 50, wherein the score information is rendered on the at least one first display as at least one interactive overlay.

52. The method of claim 50, wherein: rendering, on the at least one first display of the first client device, the online gaming information comprises rendering at least a portion of the online gaming information on a second portion of the at least one first display that is different from the first portion of the at least one first display.

53. The method of claim 46, wherein the first video is a first replay video of a recording relating to the first sporting event.

54. The method of claim 53, wherein: the online gaming information relates to a first sporting event or a second sporting event that is related to the first sporting event; and the at least one bet relates to the first sporting event or the sporting event.

55. The method of claim 54, wherein at least one team or at least one participant are featured in both of the first sporting event and the second sporting event.

56. The method of claim 54, wherein at least one league or at least one organization is featured in both of the first sporting event and the second sporting event.

57. The method of claim 54, wherein both of the first sporting event and the second sporting event relate to a same sport.

58. The method of claim 46, wherein in A), the first instructions transmitted to the first client device include a first address for a first media source, such that the first client device uses the first address to request and receive from the first media source the first digital content via the first video communication channel.

59. The method of claim 58, wherein in A), the first instructions transmitted to the first client device include a first event identifier (first Event ID) that corresponds to a source of the second digital content corresponding to the online gaming information.

60. The method of claim 59, further comprising: B) receiving from the first client device a first URL including the first event identifier; and C) establishing the second communication channel between the source of the second digital content and the first client device as a persistent connection based at least in part on B).

61. The method of claim 46, wherein in A), the first instructions transmitted to the first client device cause the at least one first display of the first client device to render at least some of the online gaming information as interactive content.

62. The method of claim 61, wherein in A), the first instructions transmitted to the first client device cause the first client device, when a user of the first client device interacts with the interactive content, to: launch further graphics or animations; receive additional information about the first sporting event; and/or navigate to an Internet location.

63. The method of claim 46, further comprising: B) transmitting second instructions to a second client device that includes at least one second display to cause the at least one second display of the second client device to render the first video or a second video relating to the first sporting event and render the online gaming information relating to the first sporting event, wherein the second instructions transmitted in B) cause the second client device to: receive, on a third communication channel, the first digital content corresponding to the first video relating to the first sporting event or third digital content corresponding to the second video relating to the first sporting event; render, on the at least one second display of the second client device, the first video based on the first digital content or the second video based on the third digital content received on the third communication channel; receive, on a fourth communication channel different from the third communication channel, the second digital content corresponding to the online gaming information; and render, on the at least one second display of the second client device, the online gaming information based on the second digital content received on the fourth communication channel.

64. The method of claim 63, wherein in B), the second instructions transmitted to the second client device include the first address for the first media source or a second address for a second media source, such that the second client device uses the first address or the second address to request and receive the first digital content from the first media source or the third digital content from the second media source via the third video communication channel.

65. The method of claim 64, wherein in A), the second instructions transmitted to the second client device include the first event identifier (first Event ID) that corresponds to the source of the second digital content corresponding to the online gaming information.

66. The method of claim 63, further comprising: C) transferring first real-time information relating to the first digital content to and from the first client device via a first real-time information communication channel; and D) transferring second real-time information relating to the first digital content or the third digital content to and from the second client device via a second real-time information communication channel, wherein at least one of the first real-time information or the second real-time information comprises: at least one chat message; at least one statistic; trivia; at least one poll; news or current event information; at least one photo; advertising content; an indication of a viewer joining or leaving the first digital content or the third digital content; at least one digital gift; and/or at least one sponsorship.

67. A system for controlling a plurality of viewer client devices to receive first digital content corresponding to a first replay video of a recording relating to a first sporting event and first betting information relating to the first sporting event, the system comprising: A) a control server to periodically retrieve, via the Internet, the betting information; B) at least one socket server communicatively coupled to the control server to: receive from the control server at least the first betting information; and transmit at least some of the first betting information to at least a first viewer client device of the plurality of viewer client devices via a first event information Internet communication channel between a first event socket of the at least one socket server and the first viewer client device, wherein the first event socket corresponds to the first betting information germane to the first sporting event; and C) at least one webserver communicatively coupled to the at least one socket server to transmit, to the first viewer client device: a first Internet address of a first media source to establish a first video Internet communication channel between the first media source and the first viewer client device to carry the first digital content corresponding to the first replay video of the recording relating to the first sporting event; and a first socket address of the first event socket to establish the first event information Internet communication channel to carry the first betting information.

68. The system of claim 67, wherein the first socket address transmitted by the at least one webserver to the first viewer client device includes a first event identifier (first EventID) that corresponds to the first event socket, such that the first viewer client device uses a first URL including the first event identifier (first EventID) in the first URL to connect to the first event socket.

69. The system of claim 67, wherein the first event information Internet communication channel to carry the betting information between the first event socket of the at least one socket server and the first viewer client device is established as a persistent connection.

70. The system of claim 67, wherein: in C), the at least one webserver transmits, to a second viewer client device of the plurality of viewer client devices, the first socket address of the first event socket to establish a second event information Internet communication channel between the first event socket and the second viewer client device; and in B), the at least one socket server transmits at least some of the first betting information to the second viewer client device via the second event information Internet communication channel, such that the first betting information is shared in a synchronized manner by the first viewer client device and the second viewer client device.

71. The system of claim 70, wherein the first socket address transmitted by the at least one webserver to the second viewer client device includes the first event identifier (first EventID) that corresponds to the first event socket, such that the second viewer client device uses a second URL including the first event identifier (first EventID) in the second URL to connect to the first event socket.

72. The system of claim 70, wherein in A), the control server is a single point of entry for the system to obtain the first betting information to reduce synchronization errors between the first viewer client device and the second viewer client device.

73. The system of claim 72, wherein: in A), the control server detects a change in status in the first betting information and transmits changes in the first betting information to the at least one socket server; and in B), the at least one socket server transmits the changes in the first betting information to the first viewer client device and the second viewer client device via the first event socket to provide a single synchronized update and mitigate client-by-client latency and/or synchronization issues.

74. The system of claim 70, wherein in C), the at least one webserver transmits to the second viewer client device one of: the first address of the first media source to establish a second video Internet communication channel between the first media source and the second viewer client device to receive the first digital content corresponding to the first replay video of the recording relating to the first sporting event; or a second address of a second media source to establish an alternate second video Internet communication channel between the second media source and the second viewer client device to receive second digital content relating to a second replay video of a second recording relating to the first sporting event the first sporting event.

75. The system of claim 74, wherein: the second client device is a subscriber to one of: the first media source and/or the first digital content; or the second media source and/or the second digital content; and in C), the at least one webserver transmits to the second viewer client device one of: a first identifier (first StreamID) for the first media source and/or the first digital content as at least a portion of the first address if the second client device is a subscriber to the first media source and/or the first digital content; or a second identifier (second StreamID) for the second media source and/or the second digital content as at least a portion of the second address if the second client device is a subscriber to the second media source and/or the second digital content.

76. The system of claim 67, wherein: in B), the at least one socket server further transmits first real-time information relating to the first digital content to at least the first viewer client device of the first plurality of viewer client devices via a first real-time information Internet communication channel between a first real-time information socket of the at least one socket server and the first viewer client device, wherein the first real-time information socket corresponds to the first digital content; and in C), the at least one webserver transmits to the first viewer client device a second socket address of the first real-time information socket to establish the first real-time information Internet communication channel.

77. The system of claim 76, wherein the first real-time information relating to the first digital content comprises: at least one chat message; at least one statistic; trivia; at least one poll; news or current event information; at least one photo; advertising content; an indication of a viewer joining or leaving the first digital content; at least one digital gift; and/or at least one sponsorship.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and/or structurally similar elements).

[0039] FIG. 1A is a block diagram showing respective elements of a system to facilitate live streaming of digital content (video and/or audio) from multiple broadcasters to multiple viewers, according to some inventive implementations.

[0040] FIG. 1B illustrates a display of an example client device in the system of FIG. 1A, showing various displayed content according to some inventive implementations.

[0041] FIG. 2 is a block diagram of the broadcast/viewing servers and memory storage devices shown in FIG. 1A according to some inventive implementations.

[0042] FIG. 3 is a block diagram showing additional details of the various interconnections between the respective components of the servers and memory storage devices shown in FIG. 2, according to some inventive implementations.

[0043] FIGS. 4A and 4B show a process flow diagram illustrating a broadcast media server selection algorithm according to one inventive implementation, which in some examples may be performed by one or more web servers shown in FIGS. 2 and 3.

[0044] FIGS. 5A through 5C show a process flow illustrating a media server process for the media servers shown in FIGS. 2 and 3, according to one inventive implementation.

[0045] FIG. 6 is a block diagram illustrating the selective coupling of an example viewer client device to one of the media sources of the servers and memory storage devices shown in FIGS. 2 and 3, according to some inventive implementations.

[0046] FIG. 7 is a process flow diagram illustrating a viewer stream source selection algorithm according to one inventive implementation, which in some examples may be performed by the one or more web servers shown in FIGS. 2, 3 and 6.

[0047] FIG. 8 is a block diagram showing additional details of the HLS server architecture of the servers and memory storage devices shown in FIGS. 2, 3 and 6, according to some inventive implementations.

[0048] FIGS. 9A through 9D show a process flow illustrating an HLS stream viewing process performed by the HLS server architecture shown in FIG. 8, according to one inventive implementation.

[0049] FIG. 10 illustrates some of the functionality (e.g., services and other processes) performed by the control server shown in FIGS. 2 and 3, according to one inventive implementation.

[0050] FIGS. 11A through 11C show a process flow diagram illustrating an RTMP media server scaling system service method performed by the control server of FIG. 10, according to one inventive implementation.

[0051] FIGS. 12A through 12C show a process flow diagram illustrating an RTMP CDN server scaling system service method performed by the control server of FIG. 10, according to one inventive implementation.

[0052] FIGS. 13A and 13B show a process flow diagram illustrating a stream and server watchdog service method performed by the control server of FIG. 10, according to one inventive implementation.

[0053] FIGS. 14A and 14B show a process flow diagram illustrating a check RTMP media/CDN server method performed by the control server of FIG. 10 (e.g., as part of the method shown in FIGS. 13A and 13B), according to one inventive implementation.

[0054] FIG. 15 shows a process flow diagram illustrating an event data ingress service method performed by the control server of FIG. 10, according to one inventive implementation.

[0055] FIGS. 16A and 16B show a process flow diagram illustrating a live event data monitor service method performed by the control server of FIG. 10, according to one inventive implementation.

[0056] FIGS. 17A and 17B show a process flow diagram illustrating an asynchronous task service method performed by the control server of FIG. 10, according to one inventive implementation.

[0057] FIGS. 18A and 18B show a process flow diagram illustrating a process for taking a screenshot (thumbnail) of a live stream, performed by the control server of FIG. 10, according to one inventive implementation.

[0058] FIGS. 19A and 19B show a process flow diagram illustrating a user login process according to one inventive implementation, which in some examples may be performed by a client device and facilitated by one or more web servers shown in FIGS. 2 and 3.

[0059] FIGS. 20A and 20B show a process flow diagram illustrating a mobile broadcaster stream create process according to one inventive implementation, which in some examples may be performed by a broadcaster client device shown in FIG. 1A and facilitated by one or more web servers shown in FIGS. 2 and 3.

[0060] FIGS. 21A, 21B, 21C, 21D, and 21E show a process flow illustrating a mobile broadcaster active stream process according to one inventive implementation, which in some examples may be performed at least in part by a broadcaster client device shown in FIG. 1A.

[0061] FIGS. 22A and 22B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for the processes shown in FIGS. 20A and 20B, and FIGS. 21A-21E, according to one inventive implementation.

[0062] FIGS. 23A and 23B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for a live stream RTMP media server or RTMP CDN viewer, according to one inventive implementation.

[0063] FIGS. 24A and 24B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for a live stream HLS viewer, according to one inventive implementation.

[0064] FIGS. 25A, 25B, and 25C show a process flow illustrating a mobile client live stream replay method, according to one inventive implementation.

[0065] FIGS. 26A, 26B, and 26C show a process flow illustrating an event data replay process called in the method of FIGS. 25A, 25B and 25C, according to one inventive implementation.

[0066] FIG. 27 shows a process flow illustrating a user input replay process called in the method of FIGS. 25A, 25B and 25C, according to one inventive implementation.

[0067] FIG. 28 shows a process flow relating to how a viewer client device requests stream information for a replay video, according to one inventive implementation.

[0068] FIG. 29 shows a process flow for an RTMP viewer, modified with respect to FIGS. 23A and 23B, based in part on the process of FIG. 28, according to one inventive implementation.

[0069] FIG. 30 shows a process flow for an HLS viewer, modified with respect to FIGS. 24A and 24B, based in part on the process of FIG. 28, according to one inventive implementation.

[0070] FIG. 31 shows a process flow for a related event selection algorithm, according to one inventive implementation.

[0071] FIGS. 32A, 32B and 32C illustrate a replay process similar to that shown in FIGS. 25A, 25B and 25C, to allow a viewer client device to replay a recorded video stream as a replay video, according to one inventive implementation.

[0072] FIGS. 33A and 33B show a process flow illustrating an event data process called in the process of FIGS. 32A, 32B and 32C, according to one inventive implementation.

[0073] FIG. 34 is a screenshot of a client device showing features relating to changing events during a stream of digital content, according to one inventive implementation.

[0074] FIG. 35 is a block diagram of the broadcast/viewing servers and memory storage devices shown in FIG. 1A, according to another inventive implementation relating to ingress of topic information.

[0075] FIG. 36 is a flow diagram illustrating a topic data ingress service method, according to one inventive implementation.

[0076] FIG. 37 illustrates a display of an example client device showing various displayed content relating to online gaming information associated with a live stream of a sporting event, according to some inventive implementations.

[0077] FIG. 38 shows a process flow diagram illustrating an online gaming data ingress service method according to one inventive implementation.

[0078] FIG. 39 show a process flow diagram illustrating a live or priority online gaming information data monitor service method according to one inventive implementation.

DETAILED DESCRIPTION

[0079] Following below are more detailed descriptions of various concepts related to, and implementations of, inventive systems, methods and apparatus for scalable low-latency viewing of broadcast digital content streams of live events, and synchronization of event information with viewed streams, via multiple Internet channels. It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in various manners, and that examples of specific implementations and applications are provided primarily for illustrative purposes.

[0080] I. Overview

[0081] The present disclosure describes inventive systems, apparatus, and methods for connecting followers of live events (e.g., sports, performances, speeches, etc.), including commentators, spectators, and/or participants in live events (e.g., athletes, performers, politicians, etc.). In some example implementations, the inventive systems, apparatus and methods further provide a social platform for sharing and contributing multimedia associated with live events.

[0082] Live streaming is used herein to refer to delivery and/or receipt of content in real-time, as events happen, or substantially in real time, as opposed to recording content to a file before being able to upload the file to a media server, or downloading the entire file to a device before being able to watch and/or listen to the content. Streaming media is used herein to refer to multimedia (e.g., digital video and/or audio media) that is delivered between two or more network-connected devices in real time or substantially in real time. Streaming may apply to continuously updated media content other than video and audio including, but not limited to, a live ticker, closed captioning, and real-time text. An end-user (e.g., a viewer) may watch and/or listen to media streamed over a network (e.g., the Internet) using a user output interface such as a display and/or over a speaker communicatively coupled with, for example, a desktop computer, notebook or laptop computer, smart television, set-top box, Blu-Ray™ player, game console, digital media player, smartphone (e.g., iOS or Android), or another network-connected interactive device.

[0083] In some implementations, a network platform receives and provides multimedia (e.g., digital video content and/or digital audio content) associated with a live event. The multimedia may be captured by one or more broadcasters present at the live event. A broadcaster present at the live event may stream video and/or audio content to the network platform in real time or substantially in real time during the live event. For example, a broadcaster may capture video of a sporting event, such as a local high school football game, using a video camera, smartphone camera, etc. The video may include audio and/or visual commentary from the broadcaster. One or more viewers (either present or not present at the event) may stream video and/or audio of the event to watch and/or listen in real time or substantially in real time during the live event to the broadcaster's commentary. Alternatively, a broadcaster present at the live event may record video and/or audio content for delayed streaming or uploading to the network platform during or after the live event, and a viewer may download the broadcaster's recording of the live event and the video and/or audio commentary for delayed viewing and/or listening.

[0084] In some implementations, a broadcaster may or may not be present at a live event to still generate multimedia content (broadcaster commentary) associated with the event during the event. For example, a broadcaster may generate audio or visual content about the event while simultaneously following the event via a live broadcast by a third party (e.g., television, radio, Internet, etc.). The multimedia content may or may not include or be integrated with video and/or audio from the event itself.

[0085] In some implementations, a network platform is capable of integrating user-generated (broadcaster-generated) multimedia with real-time data (e.g., “event information”) collected by the user or a third party. For example, a live competitive event may be integrated with scores for the event. Other real-time data may include but is not limited to alerts, statistics, trivia, polls, news, broadcaster and/or viewer messages, and/or advertising associated with or relevant to the event, a participant in the event, a location of the event, a date/time of the event, etc. In one implementation, a network platform allows a user to select content, for example, news articles, and create onscreen elements for simultaneous viewing of the content.

[0086] Audio and/or visual indications and content may be integrated with user-generated multimedia for simultaneous presentation. The presentation may be in real-time or substantially in real-time. For example, audio indications may be presented with digital video media, and/or visual content may be presented with digital audio media. In some implementations, audio and/or visual indications and content are presented simultaneously with digital audio and/or video media using multiple tracks and/or display frames or overlays. For example, digital video media of a basketball game or of a broadcaster providing play-by-play audio commentary for the game may be displayed with an overlay of a real-time scoreboard and/or ticker. Alternatively, the real-time scoreboard and/or ticker may be presented in a separate frame.

[0087] Audio and/or visual indications and content may be modifiable and/or interactive. For example, traditional news and sports broadcasting may insert audio and/or visual indications and content into an outgoing digital audio and/or video media stream. The receiving client devices have been assumed to be “dumb,” that is, only capable of displaying the audio and/or video media as received. In contrast, in inventive implementations disclosed herein “smart” client devices allow audio and/or visual indications and content to be rendered on the client side, which allows for real-time modification and interaction with viewers and/or listeners. That is, client-side rendering allows for interactivity with elements and enhanced features not available to traditional broadcasting.

[0088] FIG. 1A is a block diagram of a system according to one inventive implementation, including multiple client devices (e.g., broadcaster client devices 100A and 100B, viewer client devices 200A, 200B, 200C and 200D), broadcast/viewing servers and memory storage devices 1000 (e.g., serving as the network platform noted above), an event information provider 55, one or more news feeds (RSS feeds) 65, and a digital distribution platform (app store) 75 all communicatively coupled via the Internet 50. Each of the client devices 100A, 100B, 200A, 200B, 200C, 200D may download from the digital distribution platform 75 an app or software program that becomes resident on the client device (i.e., a client app) and performs at least some of the various broadcaster and viewer functionality described herein in connection with broadcasting live streams of digital content and viewing copies of broadcasted live streams, exchanging chat messages amongst broadcasters and one or more viewers, logging system events and providing system event messages to broadcasters and viewers, collecting and maintaining/updating event information and providing event information to broadcasters and viewers in a synchronized manner, providing and updating various animation and special effects graphics, and replaying of recorded streams.

[0089] Although FIG. 1A illustrates two broadcaster client devices and four viewer client devices, it should be appreciated that various numbers of client devices (broadcaster client devices and viewer client devices) are contemplated by the systems, apparatus and methods disclosed herein, and those shown in FIG. 1A are for purposes of illustration. More specifically, a given broadcaster may have virtually any number of viewers using respective viewer client devices to receive copies of the broadcaster's live stream of digital content via the servers and memory storage devices 1000; similarly, the system may accommodate virtually any number of broadcasters providing live streams of digital content to the servers and memory storage devices 1000, wherein each broadcaster has multiple viewers receiving copies of the broadcaster's live stream of digital content. In the example shown in FIG. 1A, a first broadcaster client device 100A provides a first live stream of digital content 102A, and a first plurality of viewer client devices 200A and 200B (grouped by a first bracket) receive respective copies 202A and 202B of the first broadcaster's live stream of digital content. Similarly, a second broadcaster client device 100B provides a second live stream of digital content 102B, and a second plurality of viewer client devices 200C and 200D (grouped by a second bracket) receive respective copies 202C and 202D of the second broadcaster's live stream of digital content. With respect to events or news that may be germane to a given broadcaster's live stream of digital content, the broadcast/viewing servers and memory storage devices 1000 may retrieve various event information from the event information provider 55 (e.g., STATS LLC), and various news from news feeds (RSS) 65, and in turn convey various event information and/or news to one or more client devices.

[0090] As discussed in further detail below, a variety of digital content format and transmission protocols are contemplated herein for the broadcaster live streams 102A and 102B output by the broadcaster client devices 100A and 100B respectively, as well as the copies of the live streams 202A, 202B, 202C and 202D received by respective viewer client devices 200A, 200B, 200C and 200D. For example, the first broadcaster client device 100A may be a mobile broadcaster client device (e.g., a smartphone) and output a live stream of digital content 102A having an H.264 MPEG-4 Advanced Video Coding (AVC) video compression standard format, via real time messaging protocol (RTMP) transport for continuous streaming over the Internet (e.g., via a persistent connection to a first media server of the servers and memory storage devices 1000). The second broadcaster client device 100B may be a web-based device (e.g., a desktop computer) and output a live stream of digital content 102B having a VP8 video compression format, transmitted via the web real-time communication (WebRTC) protocol for continuous streaming over the Internet (e.g., via a persistent connection to a second media server of the servers and memory storage devices 1000). The copies of the live streams 202A, 202B, 202C and 202D may be transmitted by the servers and memory storage devices 1000 as continuous streams using RTMP or WebRTC, or using segmented and/or adaptive bitrate (ABR) protocols (e.g., Apple's HTTP Live Streaming “HLS;” Microsoft's HTTP Smooth Streaming “MSS;” Adobe's HTTP Dynamic Streaming “HDS;” standards-based ABR protocol “MPEG-DASH”).

[0091] FIG. 1B illustrates a display 250 of an example viewer client device 200A in the system of FIG. 1A, showing various displayed content according to some inventive implementations. It should be appreciated that one or more elements of the various content discussed in connection with FIG. 1B similarly may be provided on the display of a broadcaster client device. In the example of FIG. 1B, a broadcaster is providing video-based commentary relating to a live sporting event, and the display 250 of the viewer client device 200A includes various content elements including the broadcaster's video-based commentary 252, event information 254 relating to the live sporting event about which the broadcaster is providing the video-based commentary, chat messages 258 from one or more viewers consuming the broadcaster's video-based commentary, and various graphics, special effects and/or animation elements 256 (e.g., some of which are rendered in a “lower third” of the display 250).

[0092] More specifically, as shown in FIG. 1B, the client device 200A renders in the display 250 (pursuant to execution of a client app or software program) a first broadcaster's video-based commentary 252. As discussed above in connection with FIG. 1A, the first broadcaster's video-based commentary 252 is codified in a live stream of digital content 102A provided by the first broadcaster client device 100A to the servers and memory storage devices 1000, and a copy 202A of the first broadcaster's live stream is received by the viewer client device 200A from the servers and memory storage devices 1000. The display also includes event information 254 in the form of a “scorebug,” wherein the scorebug includes indicators for the teams participating in the live sporting event, score information for the live sporting event, and event status (e.g., time clock, period or quarter, etc.). In various implementations discussed in further detail below, the scorebug may be animated, may include one or more special effects graphics elements, and/or may be interactive (e.g., the viewer may press or thumb-over one or more portions of the scorebug to launch further graphics or animations, receive additional information about the live sporting event, or navigate to another Internet location to receive additional information relating to the live sporting event).

[0093] The display 250 in FIG. 1B also includes lower-third content 256 comprising additional graphics, special effects and/or animation elements which similarly may be interactive; such elements may include a broadcaster-selected title for the broadcast, as well as text commentary from the broadcaster or event-related news. Additionally, as shown in the left portion of the display 250, the display may include one or more chat messages 258 from different viewers of the broadcaster's video-based commentary, including responses from the broadcaster themselves; as seen in FIG. 1B, the chat messages 258 may include the name of the viewer, a viewer photo, and the chat message content itself.

[0094] In some implementations, the network platform provided by the servers and memory storage devices 1000 maintains user profiles for broadcasters and viewers. Each user profile may be associated with, for example, a user email address, user device, or other unique identifier. Each user profile interface (e.g., “page” such as a webpage) may include and/or be customized with content (e.g., a profile photo, descriptive text, user-generated multimedia, favorite team imagery, etc.). In some implementations, the network platform further allows for the creation of “team” profiles; for example, event participants (e.g., individuals, groups, parties, teams, bands, schools, etc.) may share a “team” profile, wherein the team profile interface (e.g., “page” such as a webpage) may aggregate relevant content (e.g., news or current events about a particular event or team, such as polls, trivia, photo galleries, etc.) and provide further opportunities for users to contribute and connect with each other. The network platform may provide user preference options to further define a team profile interface with recommendations and/or alerts specific to a particular user (e.g., to prominently feature recent activity of a particular user).

[0095] With respect to social media-related features, as noted above the network platform provides chat capabilities such that users may engage in live public and/or private chat sessions. For example, in some implementations, users may request permission (or be allowed) to send each other private and/or public messages (e.g., direct messages). Furthermore, users may be able to purchase private and/or public virtual gifts (e.g., digital images of beers, penalty flags, etc., or profile/content enhancements like ticker tape) or provide “sponsorships” for other users. Public gifts received by a user may be displayed on the user's profile and/or with his or her content.

[0096] In some implementations, users are able to publicly and/or privately comment on, rate, “like,” or otherwise indicate their opinions on live events, event-associated topics, user profiles, team profiles, and user-generated content. Users may be able to use #hashtags within their messages, chat sessions, comments, and/or other activity to link to messages, chat sessions, comments, and/or other activity happening among other users and/or teams. Users may be able to use @ symbols within their messages, chat sessions, comments, and/or other activity to tag other users, event participants, and teams.

[0097] In some implementations, a network platform provides a directory of live events. The directory interface may be presented as a listing, drop-down menu, keyword search bar, etc. The directory interface may include and/or distinguish between different categories of events. For example, the directory interface may include and/or distinguish between events that are scheduled, underway, and/or completed. The directory interface also may include and/or distinguish between different or particular types of events (e.g., live sports versus live music, baseball versus hockey, professional versus collegiate, National League versus American League, etc.); different or particular participants in the events (e.g., team, coach, athlete, owner, school, etc.); and/or different or particular locations of the events (e.g., country, region, state, county, town, district, etc.). As discussed in greater detail below, in one implementation a dedicated control server of the network platform periodically retrieves a variety of event information from one or more event information providers (e.g., for sports events, ESPN, STATS LLC), and populates a database of the network platform with information on available events so as to provide the directory of live events to a user.

[0098] In some implementations, the network platform may provide user preference options to further define an event directory interface with recommendations and/or alerts specific to a particular user. The network platform may request the location of a user or permission to access the geo-location of the user's device in order to recommend events nearby. The network platform may track and interpret patterns in the user's use of the platform to predict and recommend events specific to the user.

[0099] In some implementations, after a user selects an event, the network platform provides a directory of other users who are present at the event and/or generating media associated with the event. The directory interface may be presented as a listing, drop-down menu, keyword search bar, etc. Selection of another user from the event-specific directory allows connection to, communication with, and/or access to media generated by that user. Thus, a user is able to discover and connect with similar users. The network platform may provide user preference options to further define a user directory interface with recommendations and/or alerts specific to a particular user. For example, in some implementations, users can discover other users based in part on one or more of the location of respective users, an event about which the broadcaster is providing commentary, a title of a broadcaster's live stream, and topics or other users that have been identified (e.g., in chat messages relating to a given broadcaster's live stream and/or a particular user's profile, using #hashtags or @ symbols).

[0100] In some implementations, the popularity of an event and/or broadcaster is monitored, displayed, and/or used in real-time or substantially in real-time. For example, a number of video servers may be scaled based on demand and/or usage by client devices, including broadcasters and/or viewers. Worker servers may be used for distributed monitoring and capturing screenshots/thumbnails of video streams. In another example, client media source selection of live stream copies, such as Real-Time Messaging Protocol (RTMP) versus HTTP Live Streaming (HLS), may be based on demand and/or usage levels (e.g., number of viewers requesting copies of a given broadcaster's live stream, capacity of media servers and/or content delivery network).

[0101] II. Servers and Memory Storage Devices

[0102] Having provided an overview of the information flow and general functionality enabled by the various elements shown in FIG. 1A, additional details of the servers and memory storage devices 1000 are now discussed, with reference initially to FIG. 2.

[0103] In particular, FIG. 2 is a block diagram providing another perspective of the system shown in FIG. 1A, showing example communication connections between the broadcaster client devices 100A and 100B and the servers and memory storage devices 1000, example connections between the servers and memory storage devices 1000 and the viewer client devices 200A and 200C, and some structural details of the servers and memory storage devices 1000. Some of the broadcaster/viewer client devices that are mobile devices (e.g., smartphones) have downloaded a client app 5000 (e.g., from the digital distribution platform or app store 75 shown in FIG. 1A) which is resident in memory of the client device and executed by a processor of the client device. For purposes of simplifying the illustration, only the viewer client devices 200A and 200C explicitly show the client app 5000 resident on the client devices; it should be appreciated, however, that one or more mobile broadcaster client devices also have the client app 5000 installed thereon.

[0104] As shown in FIG. 2, in one inventive implementation the servers/memory storage devices 1000 include one or more web servers 700 (also referred to herein as a “web server pool”) that support an Applications Programming Interface (API) to facilitate communications between the servers/memory storage devices 1000 and one or more mobile broadcaster/viewer client device executing the client app 5000, and also facilitate communications to and from web-based client devices (that access the web server(s) via a web portal at a particular URL). In this role, as discussed in further detail below, much of the instructive communication between the client devices and the servers/memory storage devices 1000 occurs via the web server(s) 700. For example, it is via the web server(s) 700 that client devices create new live streams for broadcast and get access to media servers, receive access to view other broadcasters' live streams via one of multiple different media sources, receive event information associated with broadcasters' live streams and send and receive chat messages, log on and create or update user profiles (or other profiles such as team profiles), and access other social media-related functionality (e.g., digital gift giving) to interact with other users. The web server(s) 700 are communicatively coupled to a memory system 400 that includes a database 420, data storage 440, and one or more memory caches 460 to store various information (e.g., user profile information, stream information, event information, recorded live streams, etc.) germane to the operation of the servers and memory storage devices 1000 and the various client devices.

[0105] The servers/memory storage devices 1000 further comprise a plurality of media sources 300 (e.g., computer servers including one or more processors, memory, and one or more communication interfaces) that receive a live stream of video-based commentary from a given broadcaster client device, and provide copies of the live stream of video-based commentary to one or more viewer client devices. As shown in FIG. 2, in one implementation the media sources 300 are communicatively coupled to the memory system 400, and may comprise one or more Real Time Messaging Protocol (RTMP) media servers 320, an RTMP Content Delivery Network (CDN) 340 (which itself includes a plurality of content delivery network servers), one or more WebRTC media servers 360, and an inventive HTTP Live Streaming (HLS) caching and amplifying server architecture 380. Additional details of the media sources 300 are discussed below in connection with FIGS. 3 through 6, and particular details of media server processes (performed by the RTMP media servers 320 and the WebRTC media servers 360) are discussed below in connection with FIGS. 5A, 5B and 5C. As also discussed below, in one implementation the web server(s) 700 select a particular media server of the media sources 300 to which a given broadcaster connects to provide the broadcaster's live stream of digital content, and the web server(s) 700 also select a particular media source of the media sources 300 to which a given viewer connects to receive a copy of a given broadcaster's live stream; further details of a broadcast media server selection algorithm and a viewer stream source selection algorithm implemented by the web server(s) 700 are provided below in connection with FIGS. 6 and 7.

[0106] The servers/memory storage devices 1000 shown in FIG. 2 further comprise a control server 500 coupled to the memory system 400, the event information provider 55, and the news feeds (RSS) 65 (e.g., via the Internet). In one aspect, the control server 500 periodically retrieves various event information from the event information provider 55 and/or news from the news feeds 65 that is germane to respective broadcasters' video-based commentary. In another aspect, the control system 500 may store at least some portion of retrieved event information and/or news in the memory system 400. More generally, as discussed below in connection with FIG. 10, the control server 500 implements a number of services/processes that govern functionality of other servers and devices in the servers/memory storage devices 1000; examples of such control system services/processes include, but are not limited to: an RTMP media server scaling process to add or remove servers from the one or more RTMP media servers 320 of the media sources 300 (see FIG. 11); an RTMP CDN server scaling process to add or remove servers from the RTMP CDN 340 of the media sources 300 (see FIG. 12); a live stream and media server watchdog process (see FIGS. 13-14); an event data ingress process (see FIG. 15); a live event data monitor process (see FIG. 16); an asynchronous task processor (see FIG. 17); and a live stream thumbnail/screenshot acquisition process (see FIG. 18).

[0107] With reference again to FIG. 2, the servers/memory storage devices 1000 further comprise one or more socket servers 600 communicatively coupled to the web server(s) 700 and the control server 500. In one aspect, the socket server(s) 600 facilitate communication, to one or more broadcaster client devices and one or more viewer client devices, of synchronized event information retrieved by the control server 500 and associated with video-based commentary relating to a particular event. In particular, one or more sockets of the socket server(s) dedicated to the particular event allow respective client devices to establish an event information channel with the socket server(s), such that the event information (e.g., in the form of “event messages”) is shared in a synchronized manner by all broadcasters/viewers following the particular event.

[0108] In FIG. 2, the socket server(s) 600 also facilitate communication, between a given broadcaster of a live stream of video-based commentary and corresponding viewers of copies of the live stream, of chat messages and/or system event information (also referred to collectively simply as “chat information”) relating to the broadcaster's live stream. In particular, one or more sockets of the socket server(s) 600 dedicated to the particular broadcaster's live stream allow respective client devices used by the broadcaster and their viewers to establish a chat/system event channel with the socket server(s), such that chat messages/system event information is shared in a synchronized manner by the broadcaster of the live stream and corresponding viewers of copies of the live stream. Chat messages sent on a given chat/system event channel may be displayed as text on all broadcaster/viewer client devices connected to the socket(s) dedicated to the particular broadcaster's live stream, whereas system event information may be received (but not necessarily displayed itself) by all client devices connected to the socket(s) dedicated to the particular broadcaster's live stream, and provide the client device with relevant data or instructions to take some action. As discussed further below, examples of the types of system event information or “system messages” that may be broadcast by the socket(s) dedicated to the particular broadcaster's live stream include, but are not limited to, indications of viewers joining or leaving a broadcast, an indication of a new follower of a broadcaster, indications relating to the purchase of digital gifts and types of digital gifts (which may cause some display or audio event on the client device), indications relating to “likes” (e.g., cheers, handclaps, or applause icons, or audio of crowds cheering), and other data/instructions relating to various social networking functionality.

[0109] In one aspect, connections between a given client device and a particular socket of a socket server are persistent authenticated connections (e.g., with IP-based fingerprint identifiers for anonymous users). The authenticated connection allows the servers and media storage devices 1000 to track how many users are connected to a particular socket at any given time (and hence how many users are viewing a copy of a particular broadcaster's live stream, and/or how many users are viewing a copy of a live stream relating to a particular event). In another aspect, the various “messages” (e.g., event messages, chat messages, system messages) that are carried on the respective channels between a given client device and corresponding sockets of the socket server(s) are data packets including various event information, chat to be displayed, or system events (e.g., “new viewer,” “disconnected viewer,” “stream muted, “stream ended”).

[0110] With reference again for the moment to FIG. 1A, recall that in the example arrangement depicted in FIG. 1A a first broadcaster client device 100A provides a first live stream of digital content 102A, and a first plurality of viewer client devices 200A and 200B (grouped by a first bracket) receive respective copies 202A and 202B of the first broadcaster's live stream of digital content. Similarly, a second broadcaster client device 100B provides a second live stream of digital content 102B, and a second plurality of viewer client devices 200C and 200D (grouped by a second bracket) receive respective copies 202C and 202D of the second broadcaster's live stream of digital content. Turning now again to FIG. 2, and taking only the viewer client devices 200A and 200C into consideration for purposes of illustration, the example implementation shown in FIG. 2 contemplates that the first broadcaster is providing video-based commentary about a first live sporting event, and the second broadcaster is providing video-based commentary about a second (different) live sporting event, such that the first viewer client device 200A receives the copy 202A of the first broadcaster's live stream of digital content 102A relating to the first sporting event (and provided by the first broadcaster client device 100A), and that the second viewer client device 200C receives the copy 202C of the second broadcaster's live stream of digital content 102B relating to the second sporting event (and provided by the second broadcaster client device 100B). Also, in the example of FIG. 2, the first broadcaster's live stream 102A is an RTMP stream received by the RTMP media server(s) 320, and the second broadcaster's live stream 102B is a WebRTC stream received by the WebRTC media server(s) 360. The media sources 300 provide the copy 202A of the first broadcaster's live stream 102A to the first viewer client device 200A via a first video Internet communication channel 204A, and provide the copy 202C of the second broadcaster's live stream 102B to the second viewer client device 200C via a second video Internet communication channel 204C (further details of the role of the web server(s) 700 in selecting a particular media source of the media sources 300 to which each viewer client device connects to establish a video Internet communication channel are discussed below in connection with FIGS. 6 and 7).

[0111] In the example of FIG. 2, as noted above the control server 500 periodically retrieves, via the Internet and from the event information provider 55, first event information 502A germane to the first live sporting event, wherein the first event information includes at least first score information 504A for the first live sporting event. The control server further retrieves second event information 502B germane to the second live sporting event, wherein the second event information includes at least second score information 504B for the second live sporting event. The control server passes at least the first score information 504A and the second score information 504B to the socket server(s) 600. In turn, the socket server(s) 600 establish one or more first event sockets 602A dedicated to the first event information and one or more second event sockets 602B dedicated to the second event information.

[0112] As discussed further below, the web server(s) 700 provide to the first viewer client device 200A a first event identifier (a first EventID) that corresponds to the first event socket 602A; the web server(s) 700 also provide to the second viewer client device 200C a second event identifier (a second EventID) that corresponds to the second event socket 602B. The first viewer client device 200A uses the first EventID to connect to the first event socket 602A (e.g., via a first URL including the first EventID in a path of the URL), and the second viewer client device 200C uses the second EventID to connect to the second event socket 602B (e.g., via a second URL including the second EventID in a path of the URL). The first score information 504A is then transmitted to the first viewer client device 200A via a first event information Internet communication channel 206A between the first event socket 602A and the first viewer client device 200A, and the second score information 504B is transmitted to the second viewer client device 200C via a second event information Internet communication channel 206C between the second event socket 602B and the second viewer client device 200C.

[0113] In a manner similar to that described above in connection with the first and second event information, in the example of FIG. 2 chat messages and other system event information (“chat information”) may be distributed to viewers of each broadcaster via respective dedicated sockets of the socket server(s) 600. In particular, the socket server(s) 600 similarly establish one or more first chat/system event sockets 604A dedicated to the first broadcaster's live stream of digital content 102A and one or more second chat/system event sockets 642B dedicated to the second broadcaster's live stream of digital content 102B. The web server(s) 700 provide to the first viewer client device 200A a first stream identifier (a first StreamID) that corresponds to the first chat/system event socket 604A; the web server(s) 700 also provide to the second viewer client device 200C a second stream identifier (a second StreamID) that corresponds to the second chat/system event socket 604B. The first viewer client device 200A uses the first StreamID to connect to the first chat/system event socket 604A (e.g., via a first URL including the first StreamID in a path of the URL), and the second viewer client device 200C uses the second StreamID to connect to the second chat/system event socket 604B (e.g., via a second URL including the second StreamID in a path of the URL). The first chat information 210A is then transmitted to the first viewer client device 200A via a first chat/system event Internet communication channel 208A between the first chat/system event socket 604A and the first viewer client device 200A, and the second chat information 210B is transmitted to the second viewer client device 200C via a second chat/system event Internet communication channel 208C between the second chat/system event socket 604B and the second viewer client device 200C.

[0114] For purposes of simplifying the illustration in FIG. 2, the broadcaster client devices 100A and 100B are shown only providing respective live streams 102A and 102B directly to different media servers 320 and 360; however, it should be appreciated that the broadcaster client devices 100A and 100B have additional communication connections to the socket server(s) 600 and the web server(s) 700, similar to those shown in FIG. 2 between the example viewer client devices 200A and 200C and the socket server(s) 600 and web server(s) 700, so that the broadcaster client devices may similarly receive event information and chat information on different communication channels respectively dedicated to the event information and chat information.

[0115] In view of the foregoing, it may be appreciated from FIG. 2 that, in one example implementation, there are three different communication channels between a given broadcaster/viewer client device and the broadcast/viewing servers and media storage devices 1000, namely: 1) a video communication channel (e.g., 204A, 204C) between the client device and the media sources 300 to receive a copy of a broadcaster's live stream of digital content; 2) an event information communication channel (e.g., 206A, 206C) between the client device and one or more particular sockets of the socket server(s) 600 dedicated to a particular event; and 3) a chat/system event communication channel (e.g., 208A, 208C) between the client device and one or more particular sockets of the socket server(s) 600 dedicated to a particular broadcaster's live stream of digital content.

[0116] In the example of FIG. 2, the first and second broadcasters provide to their respective viewing audiences video-based commentary regarding different live sporting events. However, as discussed elsewhere in this disclosure, it should be appreciated that the events about which the broadcasters provide video-based commentary are not limited to live sporting events, but may relate to a wide variety of other events, news, and/or particular topics of interest. Additionally, it should be appreciated that the first and second broadcasters (and additional broadcasters) may provide to their respective viewing audiences video-based commentary about the same live event; in this case, the servers and media storage devices 1000 provide the appropriate connectivity such that viewers of the same live event may effectively switch between different broadcasters' video-based commentary about the event, participate in different chat information exchanges associated with each broadcaster's live stream, and all share the same event information in a synchronized manner.

[0117] In particular, with reference again to the example of FIG. 2, consider an implementation in which both the first broadcaster's live stream of digital content 102A and the second broadcaster's live stream of digital content 102B include the broadcasters' respective video-based commentary about the first live sporting event. In this situation, the web server(s) 700 would provide to both the first viewer client device 200A and the second viewer client device 200C the first event identifier (the first EventID) that corresponds to the one or more first event sockets 602A of the socket server(s) 600, and both of the first viewer client device 200A and the second viewer client device 200B would use the first EventID to connect to the one or more first event sockets 602A (e.g., via a first URL including the first EventID in a path of the URL). In this manner, the first score information 504A would then be transmitted to both the first viewer client device 200A via the first event information Internet communication channel 206A between the one or more first event sockets 602A and the first viewer client device 200A, and to the second viewer client device 200C via a second event information Internet communication channel 206C between the one or more first event sockets 602A and the second viewer client device 200C. Thus, both of the viewer client devices in this scenario would receive the same event/score information for the first live sporting event in a synchronized manner from the socket server(s).

[0118] At the same time, however, the respective viewer client devices 200A and 200C would be connected to different chat/system event sockets of the socket server(s) corresponding to the different broadcasters' live streams; in particular, the web server(s) 700 would provide to the first viewer client device 200A the first stream identifier (the first StreamID) that corresponds to the first chat/system event socket 604A and provide to the second viewer client device 200C the second stream identifier (the second StreamID) that corresponds to the second chat/system event socket 604B. As discussed in the previous example, the first viewer client device 200A would use the first StreamID to connect to the first chat/system event socket 604A (e.g., via a first URL including the first StreamID in a path of the URL), and the second viewer client device 200C would use the second StreamID to connect to the second chat/system event socket 604B (e.g., via a second URL including the second StreamID in a path of the URL). The first chat information 210A would then be transmitted to the first viewer client device 200A via a first chat/system event Internet communication channel 208A between the first chat/system event socket 604A and the first viewer client device 200A, and the second chat information 210B would be transmitted to the second viewer client device 200C via a second chat/system event Internet communication channel 208C between the second chat/system event socket 604B and the second viewer client device 200C.

[0119] FIG. 3 is a block diagram showing additional details of various interconnections between the respective components of the servers and memory storage devices 1000 shown in FIG. 2, according to some inventive implementations. In the example of FIG. 3, some of the components of the servers and memory storage devices (e.g., 1000A) are hosted by a first web hosting service (e.g., Amazon Web Services AWS), while one or more other components of the servers and memory storage devices (1000B) may be hosted by a different web hosting service and/or generally accessible via the Internet. In yet other implementations, a single web hosting service may host all of the servers and memory storage devices. In addition to the various components shown in the example of FIG. 2, FIG. 3 also shows that the servers and memory storage devices 1000 may further include a transcoder server pool 800 (e.g., that may be employed for transcoding of recordings of a given broadcaster's live stream of digital content, for later replay via adaptive bitrate protocols), an asynchronous queue 850 (e.g., for queuing of various messages and instructions to be acted upon by an asynchronous task processor implemented by the control server 500), and a gateway NAS server 870 (e.g., to facilitate communications between a WebRTC media server pool and other elements of the servers and memory storage devices 1000A that may be hosted by the first web hosting service). Additionally, FIG. 3 illustrates that the database 420 may include a main database and multiple database shards, in which portions of data are placed in relatively smaller shards, and the main database acts as a directory for the database shards (in some implementations, the main database also stores some de-normalized data, for example, to facilitate cross-server searching).

[0120] III. Technological Solutions to Improve Computer Network Functionality, Increase Computer Processing Efficiency and Reduce Computer Memory Requirements

[0121] In developing the inventive systems, apparatus and methods disclosed herein, including the servers and memory storage devices 1000 shown in FIGS. 2 and 3 as well as the client app 5000 executed by mobile client devices, the Inventors recognized and appreciated multiple technological problems with conventional techniques for transmission of digital content via the Internet. As introduced above and discussed in further detail below, the Inventors have addressed and overcome these technological problems with innovative technological solutions to effectively realize the various technical features described herein. Examples of these technological solutions include, but are not limited to, improving computer network functionality (e.g., improving the speed of content transfer from broadcaster devices to viewer devices and synchronization of various content amongst multiple client devices), and improving processing efficiency of broadcaster and viewer client devices via execution of the client app 5000, while at the same time reducing memory storage requirements for the client app 5000 on the client devices.

[0122] More specifically, examples of the technological problems addressed by the inventive solutions provided by the servers and memory storage devices 1000 and client app 5000 include, but are not limited to: 1) how to provide relatively low latency copies of live streams of broadcaster digital content to multiple viewers of each of multiple broadcasters (e.g., broadcaster-to-viewer delay time on the order of ten seconds or less, or on the order of two-to-three seconds or less), and with relatively high quality and reliability (e.g., high definition HD and high bit rate, such as 2 to 5 megabits per second); 2) how to synchronize such low latency and high quality copies of broadcaster live streams of digital content with event information associated with the digital content (as well as chat information associated with a given broadcaster) amongst the multiple viewers of each broadcaster, irrespective of the number of viewers (e.g., 10 viewers, 1,000 viewers, or 10,000 viewers); 3) how to allow different classes/types of viewers (e.g., VIP users, premium subscribers, media professionals, registered users, anonymous users, web/desktop users, mobile users), and increasing numbers of viewers, to flexibly access each broadcaster's content with different live streaming formats (e.g., continuous streaming protocols such as real time messaging protocol or “RTMP,” web real-time communication or “WebRTC;” segmented protocols such as HTTP live streaming or “HLS,” HTTP Smooth Streaming or “MSS,” HTTP Dynamic Streaming or “HDS,” standards-based ABR protocol “MPEG-DASH”) and with different qualities of service; 4) how to effectively render “studio-quality” screen animations and special effects graphics (e.g., including “scorebugs” for sporting events) on displays of mobile client devices via a client app with a small memory footprint (e.g., less than 100 megabytes, such that the client app is downloadable via cellular networks); and 5) how to provide for viewing of a recording of a broadcaster's live stream as if the viewer was watching the live stream in essentially real-time (e.g., while recreating chat messages and event information updates). Various aspects of the technological solutions to these respective technological problems are discussed in turn below.

[0123] 1) Latency Considerations

[0124] With respect to latency considerations, the inventive systems, methods and apparatus disclosed herein contemplate particular parameters for the generation of a live stream of digital content by a broadcaster client device so as to induce only relatively low “client side” latency. To this end, in example implementations the client app 5000 installed and executing on a given client device selects an appropriate keyframe interval (e.g., 30 frames) for generating a broadcaster's live stream of digital content to ensure relatively low client side-induced end-to-end digital content latency.

[0125] In other aspects relating to reducing latency, particular parameters and techniques for handling live streams are contemplated for the servers and memory storage devices 1000 disclosed herein (e.g., adjusting buffer sizes and transcoder settings in media servers; employing hardware-accelerated transcoding of broadcaster live streams via graphic card processing to provide for adaptive bitrate copies of live streams). Furthermore, in some example implementations, the RTMP CDN 340 shown in FIGS. 2 and 3 comprises an innovative auto-scaling RTMP CDN server pool, coupled to a media server pool that receives live streams from respective broadcasters (e.g., either RTMP or WebRTC), to facilitate delivery of low-latency live streams to a larger number of multiple viewers. Additionally, for RTMP broadcasters, the RTMP media server(s) 320 in some implementations is/are on the same network as the RTMP CDN 340 (e.g., the RTMP media server(s) are communicatively coupled to the RMTP CDN servers as a virtual private network (VPN), see VPN 330 in FIG. 6) so as to facilitate low latency communications. For WebRTC broadcasters, although in some implementations the WebRTC media server(s) 360 may not be hosted by the same service as the RTMP CDN 340 (e.g., see FIG. 3), the WebRTC media server(s) are coupled to the RTMP CDN via high speed/low latency connections. The RTMP CDN servers essentially make further copies of transcoded live streams received from the media server (e.g., without any other processing or alteration) and pass on the respective further copies to multiple viewers (“direct pass-through amplification”). In this manner, the RTMP CDN servers introduce appreciably low latency (e.g., on the order of less than 150 milliseconds) and facilitate a significantly greater number of viewers than could be otherwise served by the media server itself. These exemplary aspects (as well as other aspects discussed in further detail below) provide for appreciably low latency introduced by the media servers and RTMP CDN (e.g., on the order of about 500 milliseconds or even less) and client-introduced digital content latency (e.g., on the order of about one-to-two seconds for continuous streaming consumers).

[0126] 2) Synchronization of Live Streams and Event Information

[0127] Yet another technical implementation challenge overcome by the inventive concepts disclosed herein relates to the display of event information updates (if present, e.g., if the broadcast is associated with an event), as well as screen animations and other special effects graphics that may be generally associated with the video and/or audio associated with a live stream, in a manner that is synchronized across multiple live streams with appreciably low latency. This is a particularly relevant consideration given that the systems, apparatus and methods disclosed herein are contemplated in some implementations as supporting multiple broadcasters providing video-based commentary for the same event, and each of these broadcasters may have multiple viewers of their broadcast—and thus, the technical challenge is to provide the same event information, and periodic updates to this event information, in a synchronized and low-latency manner to all of these broadcasters and viewers interested in following the same event. In exemplary implementations (e.g., as discussed above in connection with FIG. 2), this technical challenge is overcome with technological solutions implemented on both the client devices and the server architecture to which the client devices are communicatively coupled involving the use of multiple communication channels respectively dedicated to video/audio content from a given broadcaster, event information germane to an event about which any broadcaster may be providing video-based commentary, and chat information (chat messages and/or system event messages) shared amongst the broadcaster and their associated viewers.

[0128] In various inventive implementations disclosed herein (e.g., as introduced above in connection with FIG. 2), event information and updates to event information are provided to broadcaster client devices and viewer client devices via a socket-based “event information channel” dedicated to the event, and separate from the copy of the live stream of video-based commentary provided on a “video channel.” Thus, all viewers (and broadcasters) of the event, regardless of which live stream they may be generating or watching, connect to one or more sockets of a socket server that is/are dedicated to the event, such that all live streams relating to the event are similarly synchronized to event information and updates to same. Notably, if a viewer switches amongst different broadcasters of the same event (the viewer originally watches a first live stream from a first broadcaster of the event, and later selects a second live stream from a second broadcaster of the same event), the event information and updates to same (and any screen animations and special effects graphics that incorporate the event information) remain synchronized with all live streams from the different broadcasters, providing for a smooth second-screen experience across multiple broadcasters and viewers.

[0129] The technical challenge of displaying event information and updates to same in a synchronized and low-latency manner amongst multiple viewers is also addressed in part by using a single control server 500 in the server and memory storage devices 500 to gather and parse live event information captured in real-time. For example, for sporting events, game information may be obtained by the single control server from a dedicated third-party provider (e.g., STATS LLC, which is a sports statistics, technology, data, and content company that provides content to multimedia platforms, television broadcasters, leagues and teams, fantasy providers, and players). This single point of entry of event information into the server architecture, as provided by the control server, prevents synchronization errors inherent in network communications. Once a change in event status has been detected (e.g., if a play clock updates), the control server provides these changes to the one or more sockets dedicated to the event (to which all viewers and broadcasters of video-based commentary regarding the event are communicatively coupled), resulting in a single synchronized update to all client devices and thereby significantly mitigating client-by-client latency and/or synchronization issues.

[0130] 3) Flexible and Scalable Access to Broadcaster Content by Multiple Classes/Types of Viewers

[0131] The inventive systems, methods and apparatus disclosed herein and shown in FIGS. 2 and 3 further contemplate the ability to flexibly select the source of a copy of a broadcaster's live stream to be provided to respective multiple viewers from one of a number of possible media sources 300, namely: 1) the media server receiving the live stream in the first instance from a broadcaster (e.g., an RMTP media server 320 or a WebRTC media server 360); 2) an auto-scaling RTMP CDN server pool 340; or 3) an innovative HTTP Live Streaming (HLS) server architecture 360. Thus, multiple live stream transmission formats, protocols, and access endpoints are contemplated for different types and numbers of viewers that may receive copies of broadcasters' live streams at different bitrates and with different qualities of service. As noted above, in some implementations the web server(s) 700 implement a viewer stream source selection algorithm which selects an appropriate media source for a given viewer based on, for example, the type of user (e.g., VIP users, premium subscribers, media professionals) and the number of viewers of a particular broadcaster's live stream. Further details of viewer stream source selection for respective viewer client devices are discussed further below in connection with FIGS. 6 and 7.

[0132] Another salient element of the flexibility and scale-ability provided by the media sources 300 of the servers and memory storage devices 1000 shown in FIGS. 2 and 3 relates to the HLS caching and amplifying server architecture 360. Conventionally, as would be readily appreciated by those of skill in the relevant arts, HLS is not designed to be cacheable at the server level, and hence synchronization issues arise in connection with providing multiple HLS copies of a live stream to respective viewers. In particular, in conventional implementations, each HLS copy of the live stream is somewhere in a “window” of time (an HLS “buffer length”) relative to the original live stream (e.g., delayed from the original stream by some amount of time within an overall time window). This uncertainty results in the possibility of a first viewer of a first HLS copy of a live stream actually seeing the video content some time earlier than or later than a second viewer receiving a second HLS copy of the live stream, i.e., the respective viewers are not synchronized.

[0133] In exemplary implementations described herein, this technical problem is solved by employing an inventive HLS caching and amplifying server architecture 360, which is discussed in further detail below in connection with FIGS. 8, 9A, 9B, 9C and 9D. The HLS server architecture includes a “mother” server and one or more “child” servers, disposed between a media server and a content delivery network (CDN), in which the HLS mother server acts as a single “virtual viewer” from a given media server's perspective. Based on a single copy of an HLS file suite for a given broadcaster's live stream as provided by a media server and received by a mother caching server of the HLS server architecture, the mother server caches and passes on copies of the elements of the file suite (as requested) to one or more child servers, which in turn cache and pass on copies of the elements of the file suite to one or more geographically-distributed servers of a conventional (e.g., global) CDN (serving as an HLS CDN in tandem with the mother-child server architecture). In this manner, the mother and child servers of the HLS architecture act as caching and amplifying servers, so that identical HLS streams may be served from the HLS CDN server pool to multiple viewers of a given broadcast in a significantly narrower synchronization window than conventionally possible. In particular, in one example implementation discussed in greater detail below in connection with FIGS. 6A, 6B, 6C, and 6D, all HLS viewers receiving a copy of a broadcaster's live stream via the HLS server architecture including a mother caching server and one or more child caching servers are at most less than one HLS file segment duration out of synchronization with each other; this phenomenon is referred to herein as “viewer segment concurrency.” Based on the viewer segment concurrency provided by the inventive HLS server architecture, respective viewers of a given broadcast may be out of synchronization with one another by less than approximately one or two seconds at most.

[0134] 4) Client-Side Rendering of On-Screen Interactive Animations, Special Effects and/or Event Information

[0135] By way of background, in conventional sports broadcasting, game information (also sometimes referred to as a “scorebug”), as well as screen animations and other special effects graphics, are hard-embedded into the live stream of the game broadcast itself that is received by viewers. Unlike conventional scorebugs, screen animations, and/or other special effects graphics that are hard-embedded into live streams of a sports broadcast, in various inventive implementations disclosed herein graphics and effects are generated by the client device itself, separate from a given broadcaster's video-based commentary, and then integrated with (e.g., superimposed or overlaid on) the broadcaster's video-based commentary when rendered on the display of the client device. As shown for example in FIG. 1B, various graphics may be rendered on different portions of the display, for example, along a top or side of the display or in a “lower third” of the display.

[0136] For mobile client devices, the client app 5000 executing on the device is particularly configured to render a variety of “studio-quality” graphics while nonetheless maintaining a small file size for the client app (e.g., less than 100 megabytes, and in some instances from approximately 60-70 megabytes); this affords an exciting and dynamic broadcaster and viewer experience on mobile client devices, while still allowing the modestly-sized client app to be readily downloaded (e.g., from a digital distribution platform or “app store” 75) to a client device via a cellular network. In some implementations, maintaining a modest file size for the client app while providing high-quality graphics, animations and other special effects is accomplished in part by designing animated graphics and special effects as a series of individual frames (still-frame images) that are hard-coded in the client app, and rendering the series of individual frames on the display in a “stop-motion” style according to an animation timer set in the client device (e.g., 15 frames per second). In some implementations, “sprite sheets” may be used for graphics elements; in yet other implementations, the transparency of individual frames may be set on a pixel-by-pixel basis as may be required in some applications to provide for suitable overlay on the broadcaster's video-based commentary.

[0137] In another aspect, client-side rendering of screen animations and/or other special effects graphics allows such animations and graphics to be user-interactive; for example, a user (broadcaster or viewer) on a client device may “select” a screen animation/special effect graphic (e.g., via a touch-sensitive display screen of the client device) and launch additional graphics or initiate some other functionality on the client device.

[0138] For example, as discussed above with respect to live events about which a given broadcaster may be providing video-based commentary, event information and updates to event information are provided to broadcaster client devices and viewer client devices via a socket-based “event information channel” dedicated to the event, and separate from the copy of the live stream of video-based commentary provided on a “video channel.” Providing one or more sockets dedicated to the event information and separate from the live stream of video-based commentary provides for user-interactive features in connection with the event information, and/or the screen animations/special effects graphics incorporating the event information; for example, the user may select (e.g., thumb-over) the screen animation/special effect graphic including the event information and obtain access to additional (and in some cases more detailed) information relating to the event (e.g., a drill down on more granular event information, or a redirect to a web site or other app related to the particular event).

[0139] 5) Replay of Recorded Broadcaster Live Streams with Recreated Chat Messages and Event Information Updates

[0140] Another technical implementation challenge addressed by the technological solutions disclosed herein relates to the ability of a viewer to watch a recording of a live stream generated by a broadcaster client device (also referred to herein as a “video replay” of the live stream, or simply “replay”) as if the viewer was watching the live stream in essentially real-time (as it was being generated by the broadcaster client device), while also allowing the viewer to “seek” to different points in the video replay. In one aspect of video replay, the broadcaster themselves may assume the role of a post-broadcast viewer of the recorded broadcast.

[0141] In exemplary implementations, a technological solution for overcoming the technical implementation challenge of replaying a recorded live stream and also recreating various chat messages and event information updates (if present) as they occurred during the originally broadcast live stream is based, at least in part, on having the socket-based communication techniques act in a “fully-authenticated” fashion, for example, by dynamically creating “anonymous accounts” for non-registered or “anonymous” users. By creating such accounts for anonymous users, a replay log may be created that logs when any given viewer (as a registered user or anonymous user) joins and leaves a particular broadcast. Additionally, the replay log may include additional information, such as user-generated chat information, system messages, and event information updates, respectively synchronized with timestamps associated with the live stream as originally generated by the broadcaster client device.

[0142] During replay of a recording of the live stream, the viewer client device requests a segment of this replay log and, using the timestamps in the recording of the live stream, replays not only the digital content in the live stream but also recreates chat messages, system-related messages and event information updates (if present) in the same order and relative time of occurrence as if the viewer were watching the live stream in essentially real-time when originally broadcasted by the broadcaster. As the replay advances, the viewer client device requests additional segments of the log, keeping an in-memory buffer to smooth out any possible Internet connectivity issues. Such a replay log also allows for “seeking,” i.e., when a viewer fast forwards or rewinds; under these seeking circumstances, the viewer client device may retrieve the appropriate segment(s) of the replay log for the new viewing point, and continue to not only replay the recording of the live stream from the new viewing point but also recreate (in the same order and relative time) chat messages, system-related messages and event information updates (if present) as if the viewer were watching the live stream in essentially real-time.

[0143] Having outlined some of the various technological solutions provided by the inventive systems, apparatus and methods disclosed herein to technological problems with conventional approaches to live streaming of digital content, the discussion now turns to additional details of respective components of the servers and memory storage devices 1000 shown in FIGS. 1A, 2 and 3, as well as the functionality of the client app 5000 executed by client devices.

[0144] IV. Broadcaster Media Server Selection

[0145] FIGS. 4A and 4B show a process flow diagram 450A and 450B illustrating a broadcast media server selection algorithm according to one inventive implementation, which in some examples may be performed by the web server(s) 700 shown in FIGS. 2 and 3. As noted above, in one implementation a mobile broadcaster client device (e.g., a smartphone) outputs a live stream of digital content having an H.264 MPEG-4 Advanced Video Coding (AVC) video compression standard format, via real time messaging protocol (RTMP) transport for continuous streaming over the Internet, whereas a web-based broadcaster client device (e.g., a desktop computer) outputs a live stream of digital content 102B having a VP8 video compression format, transmitted via the web real-time communication (WebRTC) protocol for continuous streaming over the Internet.

[0146] In the process shown in FIGS. 4A and 4B, the web server(s) 700 know whether the broadcaster client device requesting access to a media server is a mobile client (H.264/RTMP) or a web-based client (VP8/WebRTC) based on header information in the communications to the web server from the client device. For mobile clients, the web server provides access to (e.g., provides the address of an endpoint for) one of the RTMP media servers 320 of the media sources 300, and for web-based clients generating VP8/WebRTC live streams of digital content, the web server provides access to one of the WebRTC media servers 360 of the media sources 300. If a web-based client is connecting via Adobe Flash or other external software, the client may be treated similarly to the process for mobile clients.

[0147] In some implementations, multiple media servers of the RTMP media servers 320 are segregated into at least one VIP media server and at least one non-VIP media server; similarly, some of the WebRTC media servers 360 are segregated into at least one VIP media server and at least one non-VIP media server. A given broadcaster may be directed to a VIP or non-VIP media server based on their user status (e.g., as a VIP user), and/or the availability of a particular server (e.g., based on available server capacity, in terms of total utilized connection bandwidth to the media server). In one aspect, to allow for some headroom in media server capacity, the “ideal capacity” of the server may be taken as approximately 60% of the true maximum capacity of the media server. If all non-VIP media servers exceed ideal capacity (but are at less than true maximum capacity), the process may send an internal administrative message (e.g., via SMS or email) to a system administrator to warn of a significant broadcaster load. In the event that no non-VIP servers are available to a given broadcaster (because all non-VIP servers are at true maximum capacity), the process displays “No Available Server” as an error message on the display of the broadcaster client device.

[0148] V. Media Server Process

[0149] FIGS. 5A through 5C show a process flow 550A, 550B, 550C, and 550D illustrating a media server process for the RTMP and WebRTC media servers 320 and 360 shown in FIGS. 2 and 3, according to one inventive implementation. These process flows include a “server monitor” process and a “video uploader” process that each of the RTMP and WebRTC media servers implements as they receive and process live streams from various broadcasters.

[0150] Regarding the “server monitor” process, a given media server periodically reports server statistics to be stored in the database 420, and queries the database to obtain a list of broadcaster streams that have been assigned to, and are connected to, the media server. For newly connected streams, the media server validates the stream information (e.g., StreamID), with the database, and if the stream is valid the media server starts a live transcoding process to provide different resolution copies of the live stream (e.g., 720p, 360p and 240p transcoded copies); in the case of a WebRTC media server, the media server also transcodes the VP8/WebRTC live stream to H.264 before providing the different resolution transcoded copies. In some implementations, the media server employs hardware-accelerated transcoding of the broadcaster's live stream (e.g., via graphic card processing) to ensure low latency of viewed transcoded copies of the live stream. The media then starts recording the highest resolution transcoded copy (e.g., 720p in the illustrated example) to provide a “raw video” recording, and notifies the database that the live stream has started and is available for viewing. Thereafter, the media server queues a first screenshot (thumbnail) for the live stream in the asynchronous queue (e.g., see 850 in FIG. 3) for processing by the control server 500 (see FIGS. 18A and 18B), and also queues push notifications to notify subscribers and followers of the broadcaster that the broadcaster is online with a live stream (e.g., by providing a StreamID to the followers/subscribers).

[0151] Thereafter, while the broadcaster continues to provide a live stream, and if there are any HLS viewers (discussed further below in connection with FIGS. 8 and 9A through 9D), the media server begins an HLS segmentation process to create and update an HLS file suite comprising an HLS playlist, HLS chunklists, and HLS file segments for each of the transcoded different resolution copies of the broadcaster's live stream. The media server process also periodically queues in the asynchronous queue (e.g., every five seconds or so) additional screenshots/thumbnails of the live stream. Once the broadcaster has ended the live stream, the media server process stops the recording of the highest resolution transcoded copy, sends out a system message on the chat/system event socket(s) corresponding to the broadcaster's live stream that the stream has ended, stops the live transcoding process, and stores the stream end time in the database 420. The media server process then also queues the upload of the “raw video” recording (the recording of the highest resolution transcoded copy) to the media server upload queue.

[0152] The video uploader process shown in FIG. 5A reads from the media server upload queue and, if there are any entries in the queue, uploads the corresponding raw video recording of the broadcaster's live stream to data storage 440 (e.g., Amazon S3) and stores the upload time to the database 420. The video uploader process also may notify a third-party transcoding service (e.g., see the transcoding server pool 800 in FIG. 3) to provide transcoded different resolution copies of the recorded video to facilitate adaptive bitrate replay for one or more viewers.

[0153] VI. Viewer Stream Source Selection

[0154] FIG. 6 is a block diagram illustrating the media sources 300 and the web server(s) 700 of the servers and memory storage devices 1000 shown in FIGS. 2 and 3, as well as the first and second broadcaster client devices 100A and 100B and one of the viewer client devices 200A, to facilitate a discussion of the selective coupling of an example viewer client device to one of the media sources, according to some inventive implementations. In tandem with FIG. 6, FIG. 7 is a process flow diagram illustrating a viewer stream source selection algorithm 702 according to one inventive implementation, which in some examples may be performed by the web server(s) 700.

[0155] As depicted representationally in FIG. 6, in one aspect the web server(s) 700 essentially serve as a controllable switch to couple the viewer client device 200A to one of an RTMP media server 320, the RTMP CDN 340 (which is communicatively coupled to the RTMP media server(s) in a virtual private network 330), a WebRTC media server 360 and the HLS serve architecture 360 to receive a copy of broadcaster's live stream of digital content. In the example of FIG. 6, the web server(s) 700 has facilitated a connection between the viewer client device 200A and the RTMP CDN 340 (as shown by the dashed line in FIG. 6). However, as discussed below, the web server(s) 700 may facilitate a connection between the viewer client device 200A and any one of the media sources 300 based at least in part on a number of viewers already receiving copies of the broadcaster's live stream. In one implementation, the database 420 stores user profiles for broadcasters and viewers, in which the user profile may include a user type (e.g., registered user, anonymous user, subscriber of one or more broadcasters, VIP user, media professional or media member, etc.); in this instance, the web server(s) 700 may facilitate a connection between the viewer client device 200A and one of the media servers 300 based at least in part on a type or status of a user of the viewer client device 200A and/or the number of viewers already receiving copies of the live stream.

[0156] More specifically, as shown in the process of FIG. 7, if the viewer client device sends a request to the web server(s) 700 to view a copy of a given broadcaster's live stream (e.g., based on a StreamID for the live stream that the viewer client device received in a push notification), and the web server(s) 700 determine that there are fewer than a first number (e.g., 10) of viewers already receiving copies of the live stream (e.g., based on a viewing count for the stream maintained in the database 420), the web server(s) provide to the viewer client device an address to connect directly to one of the RTMP media servers 320 or one of the WebRTC media servers 360 that is processing the broadcaster's live stream (depending on whether the broadcaster client device is a mobile H.264 or web-based VP8 client device). Irrespective of the number of viewers, the web server(s) 700 also provide an address to the viewer client device to connect directly to one of the media servers if a user of the viewer client device is a VIP subscriber or media professional. If however the user is not a VIP subscriber or media professional, and there are more than a first number of viewers already receiving copies of the live stream, the web server(s) provide to the viewer client device an address to connect to one of the CDN servers of the RTMP CDN 340. However, if all CDN servers of the RTMP CDN 340 are at their maximum capacity (e.g., as reflected in server statistics stored in the database), the web server(s) 700 provide an address to the viewer client device to connect to the HLS server architecture 360.

[0157] VII. HTTP Live Streaming (HLS) Server Architecture

[0158] FIG. 8 is a block diagram showing additional details of the HLS server architecture 380 of the servers and memory storage devices 1000 shown in FIGS. 2, 3 and 6, according to some inventive implementations. FIGS. 9A through 9D show a process flow illustrating an HLS stream viewing process 902A, 902B, 902C and 902D performed by the HLS server architecture 380 shown in FIG. 8, according to one inventive implementation. As some of the discussion of the HLS server architecture 380 relates to processing of a live stream at a media server, reference is made again to the media server process discussed above in connection with FIGS. 5A, 5B, and 5C.

[0159] HTTP Live Streaming (HLS) is a conventional HTTP-based media streaming communications protocol, in which a live media stream (e.g., video and accompanying audio) is divided up or “segmented” by an HLS media server into a sequence of small files that may be downloaded to a viewer client device via HTTP communications with the HLS media server, wherein each downloaded file represents one short segment or “chunk” of a copy of the live stream. As respective chunks of the copy of the live stream are downloaded and played by the viewer client device, the client device may select from multiple different alternate streams containing the same video/audio material transcoded by the media server at a variety of data rates (e.g., at different resolutions), allowing the HLS streaming session to adapt to the available data bit rate/bandwidth of the client device's connection to the HLS server. HLS connections are, by definition, not persistent connections between the HLS media server and the viewer client device, since requests for and delivery of HLS content uses only standard HTTP transactions. This also allows HLS content to be delivered to multiple viewer client devices over widely available HTTP-based content delivery networks (CDNs).

[0160] With reference again to the media server process in FIGS. 5A, 5B, and 5C, as a broadcaster's live stream is received by a media server it is cached for some amount of time (e.g., 10 to 30 seconds). The broadcaster's live stream typically includes a succession of frames at some frame rate (e.g., 30 frames/sec), and the succession of frames includes multiple “keyframes” associated with video encoding/compression. Such keyframes include the “full” content of an instant of the video, and these keyframes reset the basis of calculation (compression/estimation) for ensuing video information; in conventional video encoding/compression techniques, compressed frames between keyframes essentially include only information representing what has changed in the content between respective frames, and not the entire visual content for corresponding instants of the video. Increasing the frequency of keyframes in the stream of video frames reduces any errors that may be introduced in the compression process, as such errors would have a shorter lifespan (there would be fewer numbers of compressed frames between keyframes).

[0161] As indicated in FIG. 5B, an incoming live stream from a broadcaster and received by a media server (e.g., incoming H.264 from an RTMP broadcaster client, or VP8 from a WebRTC broadcaster client that has been transcoded to H.264) is transcoded (e.g., by the media server) to provide different resolution copies of the live stream at corresponding different bitrates (e.g., to facilitate adaptive bitrate streaming, as noted above). For example, the broadcaster's live stream may be transcoded to provide 720p, 360p and 240p different resolution copies of the live stream. As part of the transcoding process, the media server may be configured such that the keyframe interval for each transcoded copy is a predetermined value, and the keyframe interval for the transcoded copies may be the same as or different than a keyframe interval associated with the broadcaster's incoming live stream. Conventional examples of keyframe intervals that may be configured at a media server for transcoded copies of the live stream range from about 60 frames to 300 frames of video, and in some instances as high as 600 frames (at an exemplary frame rate of 30 frames/second, the associated time durations for such keyframe intervals range from two seconds for a keyframe interval of 60 frames to 10 seconds for a keyframe interval of 300 frames, and in some instances as high as 20 seconds for a keyframe interval of 600 frames).

[0162] As discussed above in connection with FIG. 5C, to implement HLS each of these different resolution copies is divided into small segments of video based in part on the keyframe interval of the copies. More specifically, the media server may be configured to set a target segment length (duration) of each segment into which the transcoded copy of the live stream is divided. An example of a conventional target segment duration for HLS is 10 seconds; however, as discussed below, in some implementations the media server is particular configured to have a significantly lower target segment duration to facilitate the functionality of the HLS server architecture 380 in processing copies of segmented live streams.

[0163] With reference again to FIG. 5C, the media server ultimately divides each copy of the live stream into respective video segments having a duration that is as close as possible to the target segment duration, with the proviso that a segment must start on and include a keyframe but may include one or more keyframes (i.e., the segment duration in practice is based on the target duration configured in the media server, some multiple of keyframes, and the frame rate of the transcoded copy). For purposes of illustration, and taking a conventional target segment duration of 10 seconds, a frame rate of 30 frames/second, and a keyframe interval of from 60 to 300 frames, each conventional 10 second HLS segment may have 1 keyframe (given a keyframe interval of 300 frames) or up to 5 keyframes (given a keyframe interval of 60 frames).

[0164] For each transcoded different resolution copy of the broadcaster's live stream, the HLS segments of the copy are stored as small files (referred to in HLS as .ts files). Thus, in an example in which there are 720p, 360p and 240p transcoded copies of the live stream, there are three sets of .ts files being generated and stored in memory at the media server as each of the copies are segmented by the media server. For each set of .ts files corresponding to a different resolution copy of the live stream, a “chunklist” is created and maintained by the media server that includes a list of pointers (e.g., relative URLs) to corresponding .ts files stored in memory; accordingly, in the example of three different resolution copies, there would be three different corresponding chunklists.

[0165] The number of pointers in a given chunklist may be referred to as the “HLS window” or “HLS buffer length,” and this HLS window/buffer length may be set as a configuration parameter for the media server. One conventional example of an HLS window/buffer length is 10 pointers to corresponding .ts files. The number of pointers in the chunklist multiplied by the duration of the HLS segment represented by each .ts file is referred to as the “HLS latency,” because a viewing client that requests an HLS copy (i.e., succession of .ts files) typically does not start downloading a first .ts file representing a video segment until the chunklist is completely populated with the set number of pointers to corresponding .ts files (the HLS window/buffer length). Given the example above of a conventional target segment duration of 10 seconds, this results in a conventional HLS latency on the order of 100 seconds. This HLS latency also may be viewed as a “buffer time” that provides for stability of the HLS stream in the event of communications issues or interruptions in network connectivity; the latency arising from the segment duration and HLS window/buffer length provides for the overall download and playback time of the .ts file segments before another chunklist is downloaded by a viewer client device, thereby mitigating potential connectivity issues that may occur between the client device and a CDN server during this buffer time (presuming that, under normal circumstances, it is quicker for the client to download a .ts file segment than it is for the client to play the segment). As new .ts files get created in the segmenting process for a given resolution copy of the live stream, the media server puts a new pointer to the newest .ts file into the corresponding chunklist and, once the chunklist is filled the first time with the set number of pointers corresponding to the buffer length, the oldest pointer gets “bumped out” of the chunklist when a new segment/pointer is generated, in a first-in-first-out (FIFO) manner.

[0166] Below is an example of a chunklist that includes six pointers to corresponding .ts files representing HLS video segments:

TABLE-US-00001 ==> curl −v https://we109.media.castr.live/t1/ngrp:397965_all/chunklist_w1844413579_b2096000.m3u8 * Trying 198.204.252.202... * TCP_NODELAY set * Connected to we109.media.castr.live (198.204.252.202) port 443 (#0) * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: *.media.castr.live * Server certificate: Go Daddy Secure Certificate Authority - G2 * Server certificate: Go Daddy Root Certificate Authority - G2 > Get /t1/ngrp:397965_all/chunklist_w1844413579_b2096000.m3u8 HTTP/1.1 > Host: we109.media.castr.live > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 200 OK < Accept-Ranges: bytes < Server: WowzaStreaming Engine/4.7.0.01 < Cache-Control: no-cache < Access-Control-Allow-Origin: * < Access-Control-allow-Credentials: true < Access-Control-Allow-Methods: OPTIONS, GET, POST HEAD < Access-Control-Allow-Headers: Content-Type, User-Agent, If-Modified-Since, Cache-Control, Range < Date: Thu, 08 Jun 207 21:10:47 GMT < Content-Type: application/vnd.apple.mpegurl < Content-Length: 368 < #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION: 4 #EXT-X-MEDIA-SEQUENCE:349 #EXTINF:2.352, media_w1844413579_b2096000_349.ts #EXTINF:2.04, media_w1844413579_b2096000_350.ts #EXTINF:2.002, media_w1844413579_b2096000_351.ts #EXTINF:2.001, media_w1844413579_b2096000_352.ts #EXTINF:2.001, media_w1844413579_b2096000_353.ts #EXTINF:2.001, media_w1844413579_b2096000_354.ts

[0167] In addition to a chunklist for every different resolution copy of the broadcaster's live stream, the media server also creates an HLS “playlist” file (e.g., having a file extension .m3u8) corresponding to the broadcaster's live stream. The HLS playlist includes a list of the transcoded different resolution copies of the live stream, and for each item in the list the playlist also includes a corresponding bandwidth/bitrate, one or more codecs for encoding the copy of the stream, and a pointer (e.g., relative address or URL) to the corresponding chunklist: [0168] 720p copy-bitrate-codec(s)-relative URL1 for chunklist1 [0169] 360p copy-bitrate-codec(s)-relative URL2 for chunklist2 [0170] 240p copy-bitrate-codec(s)-relative URL3 for chunklist3
An example of an HLS playlist file is provided below:

TABLE-US-00002 ==> curl −v https://we109.media.castr.live/t1/ngrp:397965_all/playlist.m3u8 * Trying 198.204.252.202... * TCP_NODELAY set * Connected to we109.media.castr.live (198.204.252.202) port 443 (#0) * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate : *.media.castr.live * Server certificate: Go Daddy Secure Certificate Authority - G2 * Server certificate: Go Daddy Root Certificate Authority - G2 > Get /t1/ngrp:397965_all/playlist.m3u8 HTTP/1.1 > Host: we109.media.castr.live > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 200 OK < Accept-Ranges: bytes < Access-Control-Expose-Headers: Date, Server, Content-Type, Content-Length < Server: WowzaStreaming Engine/4.7.0.01 < Cache-Control: no-cache < Access-Control-Allow-Origin: * < Access-Control-allow-Credentials: true < Access-Control-Allow-Methods: OPTIONS, GET, POST, HEAD < Access-Control-Allow-Headers: Content-Type, User-Agent, If-Modified-Since, Cache-Control, Range < Date: Thu, 08 Jun 207 21:07:51 GMT < Content-Type: application/vnd.apple.mpegurl < Content-Length: 368 < #EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:BANDWIDTH=2296000,CODECS=“avc1.77.41,mp4a.40.2”,RESOLUTION=1280x720 chunklist_w1844413579_b2096000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1031000,CODECS=“avc1.77.31,mp4a.40.2”,RESOLUTION=640x360 chunklist_w1844413579_b946000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=449000,CODECS=“avc1.66.30,mp4a.40.2”,RESOLUTION=426x240 chunklist_w1844413579_b414000.m3u8 * Curl_http_done: called premature == 0 * Connection #0 to host we109.media.castr.live left intact

[0171] Thus, the HLS “file suite” corresponding to a broadcaster's live stream includes: [0172] A playlist of different resolution copies with corresponding pointers to chunklists [0173] The chunklists, each containing a set of pointers to corresponding .ts files [0174] The .ts files pointed to in the chunklist for each different resolution copy

[0175] To play an HLS copy of a live stream, the viewer client device first requests a copy of the corresponding HLS playlist file from the media server. Based on the available bandwidth between the viewer client device and the media server at any given time, once the playlist is received the viewer client device selects the most appropriate resolution copy from the playlist having a bit rate that may be accommodated by the available bandwidth; this provides for adaptive bit rate streaming in that, from time to time, the viewer client device may select a different resolution/different bitrate copy of the live stream from the list of copies in the HLS playlist based on changes in the available bandwidth (e.g., quality of connection) between the viewer client device and the media server. Once the desired copy is selected from the playlist based on available bandwidth, the viewer client device then requests from the media server the current chunklist associated with the selected copy of the live stream, based on the corresponding pointer to the chunklist that is present in the playlist. As noted above, the chunklist for each copy of the live stream is continuously updated by the media server (FIFO) as new .ts files are created by the media server. Once the viewer client device retrieves the chunklist, it can then in turn begin retrieving the respective .ts files pointed to in the chunklist (e.g., via corresponding relative URLs) and playing the video segments represented in the .ts files. The viewer client device repeatedly requests the appropriate chunklist from the media server (e.g., after every video segment is played) to retrieve a current version of the chunklist. In the foregoing manner, as noted earlier, data/files are transmitted from the media server to the viewer client device upon request pursuant to HTTP, as opposed to streaming data continuously between the media server and the viewer client device via a persistent data connection.

[0176] Conventionally, for every request from a viewer that a media server receives for an HLS copy of a live stream, the media server creates a new HLS file suite for the requester, including an HLS playlist, associated chunklists, and sets of .ts files. Typically, such requests for an HLS copy of a live stream would arrive at the media server from respective (e.g., geographically distributed) servers of a CDN that are in turn communicating with respective (e.g., geographically distributed) viewer client devices. As HLS viewer demand increases for copies of a particular broadcaster's live stream, the load (e.g., CPU demand) on the media server increases based on the media server's process for generating a new HLS file suite for each new HLS requester.

[0177] Moreover, given that different viewer client devices may be requesting (via corresponding different CDN servers) an HLS copy of the live stream at different points in time, the Inventors have recognized and appreciated that significant synchronization issues arise amongst respective viewers based at least in part on the media server's process for generating a new HLS file suite for each new request. More specifically, because the media server creates different HLS file suites at different times for different requesters, a first requester viewing a first copy of the live stream likely sees the video content some time earlier than or later than a second requester viewing a second copy of the live stream, because at any given time the respective requesters may be downloading and playing different video segments from their respective chunklists. For conventional HLS applications, this lack of synchronization amongst respective viewers typically would not pose any problems in viewer experience.

[0178] However, the Inventors have recognized and appreciated that in the example context of multiple viewers viewing respective copies of a broadcaster's live stream of video-based commentary regarding a live event, and also receiving and displaying event information as real-time updates about the event, this lack of synchronization amongst respective HLS viewers may significantly and adversely impact viewer experience. For example, particularly in the context of a “second screen experience,” two different HLS viewers watching the same event on a first screen and watching the same broadcaster's live video-based commentary on a second screen may see the broadcaster's video-based commentary significantly out of synchronization with the live event on the first screen, and may receive and display event information (e.g., event score updates) on the second screen that are noticeably out of synchronization with the live event and/or the broadcaster's video-based commentary. Furthermore, if both of the viewers happen to be watching the same event together at the event venue on the same first screen (e.g., together in the same room at a gathering or party), they may find that their respective copies of the broadcaster's video-based commentary are noticeably out of synchronization on their respective viewer client devices.

[0179] In view of the foregoing technical problems relating to HLS viewer synchronization and media server loading, the Inventors have implemented an inventive technical solution via an HLS server architecture 380 that provides caching and amplifying functionality to address the above-noted technical problems. An example of such an HLS server architecture is shown in FIG. 8 and discussed in detail below, and FIGS. 9A through 9D illustrate flow diagrams that outline the process by which a given viewer client device requests and receives an HLS copy of a broadcaster's live stream via the HLS server architecture shown in FIG. 8.

[0180] In considering the various HLS multiple-viewer synchronization issues that are addressed by the HLS server architecture shown in FIG. 8 and the processes outlined in FIGS. 9A through 9D, the Inventors also have considered and addressed the overall latency implications of conventional HLS stream delivery in light of the inventive HLS server architecture disclosed herein. To this end, the Inventors have considered unconventional settings (e.g., at the media server) for various parameters relating to HLS streams such as keyframe interval, target segment duration, and HLS window/buffer length for chunklists. Recall in the discussion above that conventional examples of these parameters respectively include a keyframe interval of from 60 to 300 frames, a target segment duration of 10 seconds, and an HLS window/buffer length of 10 .ts files or “chunks,” giving rise to a conventional HLS latency on the order of 100 seconds. Such a latency is practically untenable in the example context of multiple viewers viewing the live event itself in person or on a first screen, viewing respective HLS copies of a broadcaster's live stream of video-based commentary regarding the live event as a second screen experience (which would be 100 seconds out of synchronization with the live event/first screen), and also receiving and displaying on the second screen event information as real-time updates about the event (which would be 100 seconds out of synchronization with the video-based commentary on the second screen).

[0181] The Inventors have recognized and appreciated that the above-mentioned parameters may be specifically selected (e.g., via configuration of the media server) to significantly reduce latency while sufficiently maintaining stability of HLS content delivery. To this end, in one example inventive implementation, the keyframe interval for transcoded copies of the live stream may be set to 30 frames (i.e., significantly fewer than 60 to 300 frames), the target video segment duration may be set to two seconds (i.e., significantly lower than 10 seconds, and such that the succession of HLS segments respectively have two keyframes each at a frame rate of 30 frames/second), and the HLS window/buffer length may be set to from four to six segments in a chunklist (as opposed to 10 chunks in a chunklist as suggested conventionally). These parameters result in a significantly reduced HLS latency of approximately 8 to 12 seconds, as compared to a conventional HLS latency on the order of 100 seconds

[0182] As shown in FIG. 8, in one implementation an HLS caching and amplifying server architecture 380 includes a “mother” server 382 and may also include one or more “child” servers 384A through 384D, disposed between a media server and an HLS CDN server pool 388, in which the HLS mother server acts as a single “virtual viewer” from a given media server's perspective. While FIG. 8 shows multiple child servers, it should be appreciated that in various inventive implementations the HLS server architecture need not have any child servers, or may only have one child server; however, the inclusion of one or more child servers in the inventive HLS server architecture facilitates enhanced scaling and reduced loading (e.g., CPU usage/bandwidth) on the mother server.

[0183] In example implementations, the HLS mother server, as well as one or more child servers, may be implemented as a customized NGINX-based caching server. Based on a single copy of an HLS file suite 375A (e.g., single playlist, associated chunklist(s), and associated .ts file segments) for a given broadcaster's live stream as provided by a media server 320/360 and received by the mother server 382 of the HLS server architecture, the mother server caches and passes on copies 375B of the elements of the file suite (as requested) to one or more child servers, which in turn cache and pass on copies 375C of the elements of the file suite to one or more geographically-distributed servers of a conventional (e.g., global) CDN (serving as an HLS CDN in tandem with the mother-child server architecture). In this manner, the mother and child servers of the HLS architecture act as caching and amplifying servers, so that identical HLS streams may be served from the HLS CDN server pool to multiple viewers of a given broadcast in a significantly narrower synchronization window than conventionally possible. In particular, in one example implementation, all HLS viewers receiving a copy of a broadcaster's live stream via the HLS server architecture shown in FIG. 8 are at most less than one HLS file segment duration out of synchronization with each other (referred to herein as “viewer segment concurrency”).

[0184] As noted above, in conventional HLS, a viewer client device does not maintain a persistent connection with an HLS media server; similarly, by default, HLS media servers do not allow caching of HLS files (e.g., playlists, chunklists and .ts files). In particular, as illustrated above in the examples of a conventional HLS chunklist and a playlist, these files respectively include an explicit instruction that prevents caching (i.e., “Cache-control: no-cache”). For different types of files, cache-control conventionally may be set for some time period that allows a file to be temporarily stored (i.e., cached) by a requesting server, after which a fresh copy of the file needs to be requested from its origin server by the requesting server; as noted above, however, caching is conventionally prohibited for HLS files by an explicit instruction in the files.

[0185] Unlike conventional HLS, in inventive implementations of the HLS server architecture shown in FIG. 8, when a first requester requests a copy of a given broadcaster's live stream the HLS mother server establishes and maintains a persistent connection to the media server (e.g., the RTMP or WebRTC media server receiving the broadcaster's incoming live stream). In this manner, as long as the broadcaster is generating the live stream, at least one requester is requesting a copy of the live stream, and no matter how many requests may be made by globally-distributed CDN servers for copies of the live stream on behalf of requesting viewer client devices, the media server only sees the load of one requester (i.e., the HLS mother server). In this capacity, the HLS media server does not have to make copies of the HLS file suite to provide for additional requesters of the broadcaster's live stream as would be required in conventional HLS; instead, the HLS mother server requests and receives a single copy of the playlist file from the media server. As discussed further below in connection with FIGS. 9A through 9D, the HLS mother server requests the single copy of the playlist file from the media server in response to a request for the playlist file made by one of the HLS child servers to the mother server. The HLS child server makes such a request to the mother server in response to a request for the playlist file made by a CDN server to the child server on behalf of a requesting viewer client device. In a manner similar to that noted above, an HLS child also may open up and maintain a persistent connection with the HLS mother.

[0186] In an example implementation, when the HLS mother server requests and receives the HLS playlist file from the media server, the HLS mother server re-writes the caching rule in the received playlist file to allow the playlist to be cached for some period of time for which a broadcaster may be expected to provide the live stream (e.g., some number of hours up to 24 hours or 86,400 seconds); in particular, the HLS mother server strips the “Cache-control: no-cache” setting from the received playlist file and replaces it with a new cache-control command having some duration of caching time. The HLS mother server then caches the revised playlist file (for the duration of the new caching time) and typically the playlist file need not be requested again from the media server. A copy of this revised playlist file with a re-written caching rule in turn is provided upon request to one or more of the HLS child servers, which in turn cached the revised playlist file and pass additional copies of the revised playlist file to one or more CDN servers so that the playlist file is ultimately provided to one or more requesting viewer client devices. Based on the re-written caching rule, each of the involved servers may store a copy of the revised playlist file for the duration of the broadcaster's live stream and need not request it again; and again, as noted above, the media server only “sees” one requesting viewer and provides one playlist, no matter how many actual viewers may be requesting a copy of the broadcaster's live stream.

[0187] More specifically, as shown in FIG. 9A, when a given viewing client device wishes to receive a copy of a broadcaster's live stream, the client device first queries a CDN server for a copy of the HLS playlist file corresponding to the broadcaster's live stream. If the CDN server has a copy of the playlist (e.g., based on a previous request from another viewer client device), the CDN server returns the playlist to the currently requesting client device. If however the CDN server does not have a copy of the revised playlist, the CDN server connects to a CDN load balancer 386 and in turn requests a copy of the revised playlist from one of the HLS child servers as determined by the load balancer.

[0188] If the HLS child server has a copy of the revised playlist (e.g., based on a previous request from a CDN server), the HLS child server returns the revised playlist to the currently requesting CDN server (which in turn passes the playlist on to the requesting viewer client device). If however the HLS child server does not have a copy of the revised playlist, the HLS child server requests a copy of the revised playlist from the HLS mother server.

[0189] If the HLS mother has a copy of the revised playlist (e.g., based on a previous request from one of the HLS child servers), the HLS mother server returns the revised playlist to the currently requesting HLS child server. If however the HLS mother server does not have a copy of the playlist (e.g., because this is the first request for a copy of the broadcaster's live stream), the HLS mother server establishes a persistent connection with the appropriate media server (e.g., based on the relative URL for the HLS copy of the stream at a given media server), requests a copy of the playlist, and re-writes the caching rule for the playlist as discussed above. The HLS mother then caches the revised playlist, returns the revised playlist to the currently requesting HLS child server. The child server in turn caches the revised playlist and passes the revised playlist on to the requesting CDN server, which in turn also caches the revised playlist and passes the revised playlist on to the requesting viewer client device.

[0190] As shown in FIGS. 9A through 9D, once the viewer client device has the playlist, it selects from the playlist the appropriate resolution copy of the live stream based on the associated bitrate of the copy and the available bandwidth between the viewer client device and the CDN server. Based on the selected copy of the live stream, the viewer client device then requests from the CDN server the corresponding chunklist. In a manner similar to the request for the HLS playlist, each of the CDN server, an HLS child server, and the HLS mother server may be queried in turn for a copy of the corresponding chunklist.

[0191] However, an important distinction between the playlist and a requested chunklist relates to the “freshness” of the chunklist and the re-writing of the chunklist's caching rule by the HLS mother server. In particular, whenever the HLS mother server requests a given chunklist from the media server, the mother server re-writes the caching rule in the received chunklist file to allow the chunklist to be cached for some period of time, for example, the segment duration corresponding to a single .ts file (e.g., two seconds). In particular, the HLS mother server strips the “Cache-control: no-cache” setting from the chunklist file and replaces it with a new cache-control command having some duration of caching time (e.g., corresponding to a segment duration). In one aspect, a caching time corresponding to a segment duration is contemplated given that the chunklist does not change during this duration (and thus, any requests for the chunklist during this duration are generally unnecessary). The HLS mother server then caches the revised chunklist file (for the duration of the new caching time) and a copy of this revised chunklist file with a re-written caching rule in turn is provided upon request to one of the HLS child servers, which in turn also caches the revised chunklist and passes a copy of the revised chunklist file to a CDN server so that the chunklist file is ultimately provided to the requesting viewer client devices. Based on the re-written caching rule, each of the involved servers may cache a copy of the updated chunklist file for up to but no more than the specified caching time, which ensures that each copy of the chunklist stored on a given server is “fresh” (e.g., within one segment duration) for downloading to the requesting viewer client device, while also mitigating unnecessary resources spent on attending to requests for chunklists during a time period in which there are no changes to the chunklist. In an alternate implementation, a given child server may again re-write the caching rule for a chunklist file to prevent caching of the chunklist by a requesting CDN server (and thereby cause the CDN server to request the chunklist from the child server every time the chunklist is requested from the CDN server by a viewer client device, even if respective requests come from one or more viewer client devices within a segment duration).

[0192] Referring again to FIGS. 9A through 9D, and considering a non-limiting example implementation in which the segment duration corresponding to a .ts file is two seconds and the CDN servers maintain the same revised caching rules as the HLS mother and child servers, FIGS. 9A through 9D illustrates that when a requesting viewer client device does not have a chunklist, it requests the chunklist from a CDN server. If the CDN server does not have the chunklist, or if the chunklist cached on the CDN server is more than two seconds old (i.e., exceeds the cache time), the CDN server requests the chunklist from an HLS child server; otherwise, the CDN server returns a “fresh copy” of the chunklist to the requesting client. A similar process is repeated for the HLS child server and the HLS mother server, i.e., if the HLS child server does not have the chunklist, or if the chunklist cached on the child server is more than two second old, the child server requests the chunklist from the mother server; otherwise the child server returns a fresh copy of the chunklist to the requesting CDN server. If the HLS mother server does not have the chunklist, or if the chunklist cached on the mother server is more than two seconds old, the mother server requests the chunklist from the media server, re-writes the caching rule in the chunklist file, caches the revised chunklist file, and returns a fresh copy of the chunklist to the requesting child server (which in turn passes the fresh copy of the chunklist to the requesting CDN server and the requesting client device).

[0193] Once the requesting viewer client device has a fresh copy of the chunklist, the viewer client device begins requesting the respective .ts files or “chunks” pointed to in the chunklist. In some respects, as shown in FIGS. 9A through 9D, this process is similar to the processes outlined above for requesting the playlist and requesting one of the chunklists pointed to in the playlist. For example, the requesting viewer client device requests a chunk from a CDN server and, if the CDN server has the requested chunk (e.g., because another requesting viewer previously requested the same chunk from the same CDN server and the CDN server already has the chunk cached), the CDN server returns the chunk to the client device for playing the video segment represented in the chunk. If however the CDN server does not have the chunk cached, it requests the chunk from an HLS child server (e.g., via the CDN load balancer). A similar process is repeated for the HLS child server and the HLS mother server. If ultimately the mother server does not have the chunk cached and needs to request the chunk from the media server (e.g., because this is the first viewer request for this chunk), the mother server requests the chunk from the media server, re-writes the caching rule in the chunk file (e.g., to change the caching rule from “no-cache” to some period of time, for example one hour), caches the revised chunk, and returns a copy of the chunk to the requesting child server (which in turn passes the copy of the chunk to the requesting CDN server and the requesting client device).

[0194] Once the viewer client device has downloaded all chunks pointed to in the chunklist, it plays them in turn, deletes the current copy of the chunklist that the viewer client device has cached, and then again determines the appropriate resolution copy of the live stream to request based on the associated bitrates of the different resolution copies and the available bandwidth between the viewer client device and the CDN server. Typically, it takes less time for a client to download a chunk then to play it; accordingly, if there are network issues, the copy of the stream can keep playing on the viewer client device while it downloads new chunks. For example, if the client successfully downloaded three chunks (six seconds of video) in two seconds of wall clock time, there remains a four second buffer of video at the client device in case the fourth chunk has a delay in retrieval.

[0195] The foregoing process of requesting and receiving an appropriate fresh chunklist based on available bandwidth, and downloading and playing the chunks pointed to in the chunklist, is repeated for the duration of the broadcaster's live stream. For example, if the media server stops receiving the broadcaster's live stream, the media server may provide a message to the HLS mother server (e.g., in response to a request from the mother server for a fresh chunklist) that the live stream has been terminated; alternatively, the media server may provide an empty chunklist to the HLS media server, which essentially would ultimately terminate the iterative requesting process and the connection between the media server and the mother server would time out.

[0196] In other aspects, the HLS mother server shown in FIG. 8 monitors the current pool of media servers that may be servicing different broadcasters' live streams (e.g., as indicated in the database of the servers/memory storage devices 1000), and self-configures to provide for custom routing (e.g., via relative URLs) between a requesting CDN server and a particular media server to appropriately retrieve a requested HLS copy of a given broadcaster's live stream (i.e., via the appropriate playlist and associated chunklists and .ts files). For example, custom routing functionality of the mother server may allow the targeting of specific media servers via a single entry URL (e.g., https://hls.media.castr.live/we90/t1/ngrp:123456_all/playlist.m3u8 requests retrieval of the adaptive HLS playlist from server “we90” for stream 123456, which the mother server internally translates to https://we90.media.castr.live/t1/mgrp:123456_all/playlist.m3u8 and thereby requests the playlist from the appropriate server, for which, when received, the mother server re-writes the caching rule, caches the revised playlist, and passes on the revised playlist to a requesting child server).

[0197] As noted earlier, in some implementations the HLS CDN shown in FIG. 8 that makes requests to one or more HLS child servers may be provided as the Amazon Cloudfront CDN. In any event, the geographically-distributed servers of the CDN cache to the various elements of the HLS file suite and can serve these from a variety of geographic locations to provide a virtually infinite number of HLS viewers using only a relatively small HLS CDN pool; and, irrespective of the number of CDN servers requesting content on behalf of respective viewers, the CDN serves the content quickly, and the media server sees only a single virtual viewer as the HLS mother server. In one aspect, the different “layers” of servers in the HLS server architecture introduce some degree of latency between a given broadcaster's live stream and the viewer client devices; however, as noted above, all viewer client devices have “viewer segment concurrency,” and the overall average latency for all viewers is nonetheless significantly reduced (e.g., as compared to conventional HLS). For example, given an example chunk segment duration of two seconds, and an example HLS window/buffer length of four segments, there may be up to eight seconds of latency introduced by the HLS segmenting process and another approximately two seconds of latency introduced by the transfer of files through the HLS server architecture.

[0198] It should be appreciated that the various concepts discussed herein relating to the HLS server architecture are similarly applicable to other segmented live video streaming protocols (e.g., MSS, HDS, MPEG-DASH) for which inventive server architectures are contemplated by the present disclosure.

[0199] VIII. Control Server and Associated Services/Processes

[0200] FIG. 10 illustrates some of the functionality (e.g., services and other processes) performed by the control server 500 shown in FIGS. 2 and 3, according to one inventive implementation. As noted above, the control server 500 is coupled to the memory system 400, one or more event information providers 55, one or more news feeds (RSS) 65 or other news sources, and the socket server(s) 600. In one aspect, the control server 500 periodically retrieves various event information from the event information provider 55 and/or news from the news feeds 65 that is germane to respective broadcasters' video-based commentary. In another aspect, the control system 500 may store at least some portion of retrieved event information and/or news in the memory system 400. More generally, the control server 500 implements a number of services/processes that govern functionality of other servers and devices in the servers/memory storage devices 1000; examples of such control system services/processes include, but are not limited to: an RTMP media server scaling process to add or remove servers from the one or more RTMP media servers 320 of the media sources 300 (see FIG. 11); an RTMP CDN server scaling process to add or remove servers from the RTMP CDN 340 of the media sources 300 (see FIG. 12); a live stream and media server watchdog process (see FIGS. 13-14); an event data ingress process (see FIG. 15); a live event data monitor process (see FIG. 16); an asynchronous task processor (see FIG. 17); and a live stream thumbnail/screenshot acquisition process (see FIG. 18).

[0201] 1) Server Auto-Scaling Systems and Watchdogs

[0202] FIGS. 11A through 11C show a process flow diagram illustrating an RTMP media server scaling system service method 1102A, 1102B and 1102C performed by the control server of FIG. 10, according to one inventive implementation. In the method shown in these figures, the control servers automatically scale the number of RTMP media servers 320 of the media sources that are available for broadcasters based in part on the capacity demand for the servers (e.g., number of broadcasters providing live streams). The control server monitors various media server statistics that are maintained in the database 420 (e.g., number of active servers in the RTMP media server pool; servers marked for shutdown; individual server information such as server status active/shutdown, numbers of active connections to live streams, current capacity, date/time of when server first came online for availability, etc.) and brings servers in and out of the RTMP media server pool based at least in part on the server statistics. In various aspects, the control server maintains a minimum number of servers (e.g., at least two, or a minimum capacity corresponding to approximately double the cumulative traffic at a particular time) in the RTMP media server pool to allow for spikes in stream creation, and also provides for various buffering times to allow new servers to come online. FIGS. 12A through 12C show a process flow diagram illustrating an RTMP CDN server scaling system service method 1202A, 1202B, and 1202C performed by the control server of FIG. 10, according to one inventive implementation, that is similar in many respects to the method 1102A, 1102B and 1102C performed for the media server scaling service.

[0203] FIGS. 13A and 13B show a process flow diagram illustrating a stream and server watchdog service method 1302A, 1302B performed by the control server of FIG. 10, according to one inventive implementation. The stream watchdog performed by the control server essentially ensures that new streams created by broadcasters are valid and deletes streams that were created but not started, or that have been inactive for some period of time (e.g., 30 seconds). When streams are ended, the method generates final viewer statistics (e.g., stream duration, average number of viewers, maximum number of viewers, number of simultaneous viewers, viewers added, viewers left, etc.), broadcasts a “stream ended” system event message to the chat/system event socket(s) of the socket server(s) dedicated to the broadcaster's live stream, ends the recording of the live stream by the media server, and queues the recording to the video uploader queue of the media server process. The server watchdog portion of the method 1302A, 1302B monitors the RTMP media servers and the servers of the RTMP CDN and invokes the check RTMP Media/CDN server method 1402A, 1402B shown in FIGS. 14A and 14B. As part of the server watchdog process, for new servers the control server determines a capacity of the server (e.g., based on server type), and updates the database 420 with the capacity of respective servers, server class, launch time, status update (e.g., active and available for connections) and determines a total user/streamer capacity based on newly added servers. For servers that are already online, the server watchdog ensures that servers remain active for certain intervals (e.g., 30 second intervals), automatically removes inactive servers from the pool, and reports active server status back to the database. If servers are marked for shutdown, the server watchdog archives server statistics, removes the server from the active server list stored in the database, and determines an updated total user/streamer capacity based on the removal of the server from the active list.

[0204] 2) Event Information Ingress and Live Event Monitoring

[0205] In some inventive implementations, another significant role of the control server 500 shown in FIGS. 2, 3 and 10 relates to collecting of event information and/or news (e.g., from external Internet providers), maintaining relevant event information and/or news in the database 420 (e.g., to facilitate selection of broadcasters to follow, and/or particular broadcaster live streams to view), and distributing the collected information to multiple broadcaster and viewer client devices in a relatively low-latency and synchronized manner with respect to broadcasters' video-based commentary.

[0206] In some implementations, the technical challenge of displaying event information and updates to same in a synchronized and low-latency manner amongst multiple client devices is addressed in part by using a single control server 500 to gather and parse live event information captured in real-time. For example, for sporting events, game information may be obtained by the single control server from a dedicated third-party provider (e.g., STATS LLC). This single point of entry of event information prevents synchronization errors inherent in network communications. Once a change in event status has been detected (e.g., if a play clock updates), the control server provides these changes to the one or more sockets dedicated to the event (to which all viewers and broadcasters of video-based commentary regarding the event are communicatively coupled), resulting in a single synchronized update to all client devices and thereby significantly mitigating client-by-client latency and/or synchronization issues.

[0207] In some example implementations, the control server 500 implements two service methods relating to event information, namely, an event data ingress service and a live event data monitor service. The event data ingress service is performed with a first periodicity (e.g., once or twice a day) to maintain and update an event list in the database 420. The live event data monitor service is performed with a second and more frequent periodicity (e.g., once a minute) to check for any events that are in progress and, if found, to retrieve fresh data about an in-progress event from the event information provider (e.g., at an even greater frequency, for example once a second). Similar services may be implemented by the control server 500 to ingest news on particular topics, trending threads, etc.

[0208] FIG. 15 shows a process flow diagram illustrating an event data ingress service method 1502 performed by the control server of FIG. 10, according to one inventive implementation, and FIGS. 16A and 16B show a process flow diagram illustrating a live event data monitor service method 1602A, 1602B performed by the control server of FIG. 10, according to one inventive implementation. In these methods, an event information provider is contemplated as supporting multiple different types of events for furnishing information (various types of sporting events such as basketball, football, baseball, hockey, etc.), and providing information for each instance of an event of a given event type (e.g., information for each of multiple basketball games, each of multiple football games, each of multiple baseball games).

[0209] For each event, the control server retrieves the raw information provided by the event information provider, and in some instances converts and/or compresses the raw information to provide a standardized format of essential data elements for storing in the database 420 and/or distribution to client devices (e.g., via broadcast of event messages having the standardized format to one or more dedicated sockets of the socket server(s) 600). Examples of data elements for event information include, but are not limited to, a type of the event, an identifier for the event (EventID), a status of the event (e.g., pre-game, in-progress, final), score information for the event, team information for the event, a progress indicator or progress details for the event (e.g., quarter, period, inning, half-time; for baseball—balls, strikes, base information; for football—possession, down, yards to go; for basketball—timeouts, fouls), an event date and/or time of the event (e.g., actual or elapsed time information), and event participant data regarding participants in the event. In some examples, the control server further normalizes the event date and/or time to a particular reference frame (e.g., converting from UTC to EST/EDT).

[0210] In the process 1602A and 1602B shown in FIGS. 16A and 16B, the control server particularly queries the event information provider for a list of all events in a particular window around the current time (e.g., a 48 hour window, for events with start times 24 hours in the past through 24 hours in the future), to allow tracking of in-progress events (or identify any events that had inconsistent or incorrect start times or late modifications to event information). For each in-progress event, an event clock and other event information (e.g., score information, other more detailed information about the event) are updated frequently (e.g., once a second) to provide regular updates of event information messages that are broadcast to one or more dedicated event information sockets of the socket server(s) 600.

[0211] 3) Asynchronous Task Processing

[0212] FIGS. 17A and 17B show a process flow diagram illustrating an asynchronous task service method 1702A, 1702B performed by the control server of FIG. 10, according to one inventive implementation. The control server periodically reads a task or task bundle from the asynchronous queue to initiate various other actions or processes in connection with the serves and memory storage devices 1000. A number of different asynchronous system events may be implemented by this process, only some examples of which are illustrated in FIGS. 17A and 17B. For example, if an entry in the queue relates to a “Stream Started” system event, the asynchronous task processing sends out push notifications (including a StreamID) to followers and subscribers of the stream's broadcaster. Another system event processed by the asynchronous task process is when there is a new follower of a broadcaster's stream (“newFollowingStream”), for which the process loads user data and stream data, and attends to various user notifications as appropriate (e.g., email notifications, web push notifications). The asynchronous task processor is also responsible, in some implementations, for taking periodic screenshots/thumbnails of a live stream (as discussed below in connection with FIGS. 18A and 18B).

[0213] With respect to various push notifications handled by the control server 500 and/or the web server(s) 700 (or other servers of the architecture 1000), it should be appreciated that specific apps on mobile client devices need not be open for a push notification to be received on the client device. Thus the client device may receive and display social media or text message alerts even when the device's screen is locked, and/or when the app pushing the notification is closed. For iOS devices, for example, the Apple Push Notification Service API may be employed to enable the client app 5000 to receive various push notifications.

[0214] With reference again to FIG. 10, the async queue monitoring is an application that runs on the control server and that looks at the current size of the asynchronous queue and will notify an administrator. Typically, the queue of tasks to process is small (e.g., at any given second it may be between 0-10 items), and if the queue grows to a larger size (e.g., 1000 items) the async queue monitor indicates to a system administrator that there is a problem in the asynchronous task processing (e.g., additional processing resources are required, or a looping event is getting processed and re-added to the queue instead of being removed).

[0215] 4) Acquiring Screenshots/Thumbnails

[0216] FIGS. 18A and 18B show a process flow diagram illustrating a process 1802A, 1802B for taking a screenshot (thumbnail) of a live stream, performed by the control server of FIG. 10, according to one inventive implementation (in other implementations, the web server(s) 700 of other servers of the architecture 1000 may perform the process of taking thumbnails of live streams pursuant to the general technique outlined in FIGS. 18A and 18B).

[0217] With reference again to FIG. 5C and the media server process, the media server process queues to the asynchronous queue a first screenshot for a new live stream, and periodic updates to screenshots (e.g., every five seconds or so) during the duration of the live stream. These screenshot tasks are read by the asynchronous task process 1702A and 1702B discussed above in connection with FIGS. 17A and 17B and implemented by the process shown in FIGS. 18A and 18B.

[0218] In the process 1802A, 1802B, in one implementation screenshots are taken based on a broadcaster's live stream in H.264 (or transcoded to H.284 if the live stream is VP8/WebRTC from a web broadcaster). Screenshots are taken on the next available keyframe after the process is invoked. If the screenshot is not the first one taken, the stream information (e.g., in the database 420) is updated with information relating to the newest screenshot, and the screenshot is added to archived screenshots (e.g., in the data storage 440). The screenshot is then broadcast to the chat/system event socket of the socket server(s) 600 dedicated to the broadcaster's live stream.

[0219] Whenever a screenshot is taken of the broadcaster's live stream (particularly if it is the first screenshot), it may be resized for social media network requirements, and overlaid with graphics, watermarks, or promotional material. If the broadcaster requested for social share in creating the new stream (see discussion below regarding creation of new broadcaster streams), the process submits a link to the resized screenshot (e.g., an address or URL) to the indicated social network platform (e.g., Facebook, Instagram, Twitter, etc.), in some instances together with a “share graphic.” In any case, the process determines the list of users that follow/subscribe to the broadcaster, and queues a system event message (e.g., “new FollowingStream”) event for each subscriber to broadcast the first screenshot of the new live stream. As above, the stream information (e.g., in the database 420) is updated with information relating to the screenshot, and the screenshot is archived (e.g., in the data storage 440) and broadcast to the chat/system event socket of the socket server(s) 600 dedicated to the broadcaster's live stream.

[0220] With respect to sharing screenshots with social networks if elected by the broadcaster, in another implementation (not shown in FIGS. 18A and 18B), all screenshots of the broadcaster's live stream that are taken as of a given time are processed by a facial recognition algorithm to provide one of multiple options (e.g., the best of three screenshots) for selection by the broadcaster. For example, the process acquires a screenshot at 1, 3 and 5 seconds, and then every 5 seconds thereafter. The facial recognition algorithm detects candidate screenshots on rolling basis based on, for example, the clarity of the image, the quality of the face that is visible, and if the user is smiling. More specifically, every acquired screenshot is analyzed and then the “best three” are selected and presented as options to the broadcaster/viewer during social share. The broadcaster/viewer selects their preferred image, and the social share endpoint that is ultimately provided by the process to the selected social media platform(s) includes a link (e.g., address or URL) to the screenshot selected by the broadcaster.

[0221] IX. Client-Side Features (e.g., Functionality of the Client App)

[0222] Having provided various details of the servers and memory storage devices 1000 shown in FIGS. 2 and 3, attention now turns to the functionality of the client devices relating to establishing user profiles (e.g., upon login), creating broadcaster stream sessions and providing live streams from broadcaster client devices to a media server, receiving copies of a live stream at a viewer client device (e.g., from a media server, the RTMP CDN, or the HLS server architecture), providing special effects graphics and animations (including animated real-time “scorebugs”) on displays of client devices, and replaying copies of a recorded live stream from a broadcaster.

[0223] As noted earlier, unlike conventional scorebugs, screen animations, and/or other special effects graphics that are hard-embedded into live streams of a sports broadcast, in various inventive implementations disclosed herein graphics and effects are generated by the client device itself, separate from a given broadcaster's video-based commentary, and then integrated with (e.g., superimposed or overlaid on) the broadcaster's video-based commentary when rendered on the display of the client device. For mobile client devices, the client app 5000 executing on the device is particularly configured to render a variety of “studio-quality” graphics while nonetheless maintaining a small file size for the client app (e.g., less than 100 megabytes, and in some instances from approximately 60-70 megabytes); this allows the modestly-sized client app to be readily downloaded to a client device via a cellular network. In other aspects, client-side rendering of screen animations and/or other special effects graphics allows such animations and graphics to be user-interactive and/or user-customizable.

[0224] FIGS. 19A and 19B show a process flow diagram illustrating a user login process according to one inventive implementation, which in some examples may be performed by a client device and facilitated by one or more web servers 700 shown in FIGS. 2 and 3. As illustrated, a login process may be implemented by phone (via SMS message with code sent to phone, and code validation), of via a social media network platform login process (e.g., Facebook, Twitter, Instagram). For new user accounts, a user may establish a user profile that is stored in the database 420 and that may be referenced by a UserID after creation, and include a user name, profile picture, and a user status or user “type” for the user (e.g., a VIP user or member, a media professional or member of the media).

[0225] 1) Broadcaster Processes

[0226] FIGS. 20A and 20B show a process flow diagram illustrating a mobile broadcaster stream create process according to one inventive implementation, which in some examples may be performed by a broadcaster client device (pursuant to execution of the client app 5000) and facilitated by one or more web servers (700) shown in FIGS. 2 and 3. While much of the discussion above relates to an example in which a broadcaster wishes to provide a live stream of digital content including video-based commentary about a particular event, in other implementations the broadcaster may desire to create a live stream about a particular topic of interest (e.g., “anything”), or a news story, for example. For each of these options, the broadcaster may enter a title for the live stream, and the client device may request (e.g., from the web server(s) 700) a list of events or news items for selection by the broadcaster, as well as a pre-populated list of tags (as noted above, event information and/or news may be ingressed by the control server 500, and some event information and/or news may already be cached in the data cache 460 or stored in the database 420).

[0227] The broadcaster may also enter tags to be associated with the live stream to facilitate searching and various social media functionality (e.g., to allow other users to search for and find the live stream based on various criteria represented by the tags). The broadcaster may also elect other options in the stream creation process, examples of which include, but are not limited to, sharing an announcement of the stream starting on a social network platform, and enabling sharing of their location to other users (e.g., to facilitate viewing of the broadcaster's live stream by viewers based on the location of the broadcaster).

[0228] The broadcaster stream create process then submits a “stream create” request to the web server(s) 700. If the broadcaster selected a particular event from the list of events about which to broadcast, an EventID associated with the event is included in the stream create request. Other contents of the stream create request includes, but is not limited to, an API key (to authenticate the user), the title of the stream, any tags selected, newsID (if news was selected), the broadcasters social network sharing options, and broadcaster location data (if permitted by the broadcaster). The web server(s) 700 in turn validates the API key, assigns a StreamID to the newly created live stream, runs the broadcast media server selection algorithm (e.g., see FIGS. 4A and 4B) to select a media server to which the broadcaster client device connects, and returns to the broadcaster client device the StreamID and the host name (“hostname”) for the selected media server. The web server(s) 700 store in the database 420 a variety of stream information for the new live stream, which may include, but is not limited to, the StreamID, the UserID, the EventID, the DBshard, type of stream (RTMP/WebRTC), create time, hostname, title, tags, social notify options and social media platforms, location share option, location (if selected as an option) and, if the stream is associated with an EventID, an archived copy of event information at the stream create time.

[0229] FIGS. 21A, 21B, 21C, 21D, and 21E show a process flow illustrating a mobile broadcaster active stream process 2102A, 2102B, 2102C, 2102D and 2102E according to one inventive implementation, which in some examples may be performed at least in part by a broadcaster client device. In particular, the broadcaster client device accesses the media server selected by the web server(s) 700 via a particular URL (e.g., including the hostname in a path of the URL), as discussed below in connection with FIGS. 21A through 21E. The broadcaster client device then connects to a particular socket of the socket servers dedicated to the broadcaster's live stream, based in part on the StreamID provided by the web server(s), to establish a chat/system event channel. As noted above, in one aspect connections between client devices and a particular socket of a socket server are persistent authenticated connections, so that the number of users (broadcasters and viewers) connected to a particular socket (e.g., and currently watching a particular live stream and/or particular event) may be tracked. If the broadcaster's live stream is about an event, the broadcaster's client device also connects to a particular socket of the socket servers dedicated to the event, based on the EventID, to establish an event information channel.

[0230] In a “main loop” of the broadcaster client device stream active process (which for mobile clients is executed by the client app 5000), an internal frame and time clock is periodically updated, and is used for animations and special effects graphics and synchronizing of some system messages that are received via the chat/system event socket (e.g., a default chat message displayed on the client device at the beginning of each new stream that says “keep it family friendly!”). The client device then checks to see if any further system messages or chat messages are received on the chat/system event channel, and displays chat messages and/or takes other actions in response to system messages such as “member_added” (increase viewing count), “member_removed” (decrease viewing count), “new follower” (add notice to chat). Although only three system messages and corresponding actions are shown in FIG. 21B, it should be appreciated that additional and/or other types of system messages may be received on the chat/system event channel (e.g., relating to other social networking functionality, and/or digital gifts) and initiate corresponding actions as part of the stream active process.

[0231] The client device next checks to see if any event messages or data is received on the event information channel (e.g., updates to event status, event score information, event clock, other event information). The client device then captures a camera frame for the live stream and sends the frame to the media server. The client device then checks the internal frame and time clock to see if any updates are needed to animations or special effects graphics (e.g., scorebugs) to be rendered on the display of the client device (“graphics/animation layers”). In some implementations, graphics and animations are updated at a rate of between 15-25 frames/second based on the internal frame and time clock. As noted above, in some implementations for mobile client devices, animated graphics and special effects are hard-coded in the client app as a series of individual frames (still-frame images), and rendered on the display in a “stop-motion” style according to the internal frame and time clock.

[0232] In the stream active process shown in FIG. 21C, the process further queries for broadcaster input, examples of which include a request to end the stream, a request to share the stream, a request to view a list of viewers of copies of the live stream, interaction with the graphics/animations (e.g., “bottom third”), and a request to flip the camera. As also noted above, rendering graphics and animation layers on the client-side provides for user-interaction with the displayed graphics and animation layers. While not shown explicitly in FIG. 21C, as discussed above interactions with graphics/animations (“set animation state to transition to open”) may in some implementations launch a variety of other processes including, but not limited to, launching further graphics or animations, receiving additional information about the live sporting event (e.g., by thumbing-over a scorebug), or navigating to another Internet location to receive additional information relating to a live event.

[0233] In FIG. 21D, the stream active process then queries if the stream state is set to close (e.g., in response to a broadcaster's request to end the stream, discussed immediately above). If not, the process returns to updating the internal frame and time clock. If the stream state is set to close, the client device disconnects from the media server, requests final stream statistics from the chat/system event channel, and displays an end of stream screen on the display of the client device.

[0234] FIGS. 22A and 22B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for the processes shown in FIGS. 20A and 20B, and FIGS. 21A-21E, as well as the media server processes shown in FIGS. 5A, 5B and 5C, according to one inventive implementation. In essence, FIGS. 22A and 22B provide another perspective and summarize the various process flows and corresponding devices involved in the creation and provision of a live stream of digital content by a broadcaster to a media server, and the processing of the live stream by the media server. Although FIGS. 22A and 22B are directed primarily to the overall process flow for a mobile broadcaster, the functionality and devices shown in these figures applies similarly to web-based broadcasters as well.

[0235] 2) Viewer Processes

[0236] FIGS. 23A and 23B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for a live stream RTMP media server or RTMP CDN viewer, according to one inventive implementation. A viewer who is a registered or anonymous user, but has received a StreamID for a particular broadcaster's live stream (e.g., via a push notification) to their viewer client device, may send a request to the web server(s) 700 (via the API) to receive a copy of the broadcaster's live stream. The web server(s) first checks the memory cache 460 for, or requests from the database 420, various stream information corresponding to the StreamID provided by the requesting viewer. The web server(s) then perform(s) the viewer stream source selection algorithm discussed above in connection with FIG. 7 to provide an endpoint to the viewer client device for the appropriate media source from which to obtain a copy of the live stream. In the process shown in FIGS. 23A and 23B, the viewer stream source selection algorithm provides an endpoint (e.g., address or URL) to the viewer client device to establish a video communication channel with either a particular media server of the RTMP media server pool 320, or a particular server of the RTMP CDN 340.

[0237] The viewer client device also connects to the appropriate socket of the socket server(s) dedicated to the live stream to establish a chat/system event channel and thereby receive chat messages and system messages. If the live stream relates to an event, the viewer client device also connects to the appropriate socket of the socket server(s) dedicated to the event to establish an event information channel and thereby receive event messages containing various event information. The viewer using the viewer client device also may send chat messages to the web server API, which the web server directs to the appropriate socket of the socket server(s) dedicated to the live stream for broadcast to other viewers connected to the socket as well as the broadcaster. The web server also updates a replay log with the chat message from the viewer, so that the chat may be recreated if a recording of the broadcaster's live stream is replayed by a viewer at a later time (discussed further below).

[0238] FIGS. 24A and 24B show a communication flow diagram illustrating process flow elements and the server and/or memory storage devices involved in the communication flow for a live stream HLS viewer, according to one inventive implementation. The process shown in these figures is substantially similar to that outlined above in connection with FIGS. 23A and 23B; the primary difference is that, as a result of the web server(s) performing the viewer stream source selection algorithm (see FIG. 7), the web server(s) return(s) to the viewer client device an endpoint (e.g., address or URL) to establish a video channel with the HLS server architecture 380 rather than a server of the RTMP media server pool 320 or the RTMP CDN 340.

[0239] FIGS. 25A, 25B, and 25C show a process flow illustrating a mobile client live stream replay method, according to one inventive implementation. For replay of a recording of a broadcaster's live stream, the servers and memory storage devices 1000 log all events that occur in connection with a live stream (e.g., chat messages and system event messages, as well as event message) and tie them to a timestamp. This allows synchronization of all events to the replay in the same order that the events occurred during the live stream, as if the viewer were not watching a recording of the live stream but actually watching a copy of the live stream in real time.

[0240] As shown in the figures, the viewer client device couples to the web server(s) via the API to request stream information and, if the stream recording is ready, loads the initial replay data from the API and then loads the media file of the recording. The viewer client device also connects to the chat/system event socket corresponding to the live stream (via a persistent authenticated connection), not to receive chat messages or system event messages (these messages are not present on replay), but rather so that the system knows of the viewer's presence and connection. Playback of the video is then started, and then the internal clock and the current video time clock are updated to provide for appropriate buffering of the video data. As the recording is played back, event data (e.g., chat messages, system messages, event information messages) is processed in one implementation according to FIGS. 26A and 26B, and user inputs are processed in one implementation according to FIG. 27.

[0241] Replay Video with Current (Real-Time) Event/Topic Information

[0242] In another example implementation, a viewer client device may request a replay of a recording of a live stream that may be related to a particular event in the past or an ongoing event that has already started—or, the recording of the stream may be more generally related to one or more particular topics of interest. Recall that an event has a temporal component, whereas a topic of interest more generally relates to a classification of subject matter that may or may not include a temporal component (thus, all events relate to one or more topics, but not all topics are events). In any case, for purposes of the present discussion, a replay of a recording of a live stream relating to a past event, an ongoing event that has already started, or one or more topics of interest that may or may not have a temporal component, is referred to generally herein as a “replay video.”

[0243] Unlike the example replay implementation discussed immediately above in connection with FIGS. 25, 26 and 27, in the present example implementation the servers and memory storage devices 1000 may provide to the viewer client device, in tandem with the replay video, current (real-time or essentially real-time) information that is in some manner related to one or more aspects of the replay video.

[0244] Recall that, as discussed above in connection with FIGS. 20A and 20B, upon generation of a live stream of digital content, the generator of the stream (e.g., a broadcaster) may provide a title for the stream as well as one or more tags to be associated with the stream; such tags may facilitate searching and various social media functionality (e.g., to allow prospective viewers to search for and find the stream based on various criteria represented by the tags). The stream title and one or more tags for the stream are all stored in the memory system 400 (e.g., the database 420) shown in FIG. 2. For example, the web server(s) 700 may assign a StreamID to the new stream, and then store in the database 420 a variety of stream information which may include, but is not limited to, the StreamID, an EventID (if the stream is associated with an event), one or more TopicIDs (if the stream is associated with one or more topics, as discussed further below), a stream create time, the title, one or more tags, social notify options and social media platforms, and an archived copy of event information at the stream create time (if the stream is associated with an event).

[0245] In support of the present example implementation relating to replay video displayed on a viewer client device with current information relating to the replay video, FIG. 28 is a flow chart illustrating how a viewer client device generally requests stream information (e.g., from the servers and memory storage devices 1000 and, more specifically, the web server(s) 700) for a replay video. In one aspect, the process shown in FIG. 28 is a modification to the respective processes shown in FIGS. 23A and 23B (RTMP CDN Viewer Process) and FIGS. 24A and 24B (HLS Viewer Process); more specifically, FIG. 29 shows a modified RTMP CDN Viewer Process and FIG. 30 shows a modified HLS Viewer Process incorporating some of the modifications outlined in FIG. 28. Additionally, as would be appreciated throughout the discussion herein, a “data feed” or “data source” or “information source” may refer to any form of data feed (e.g., RSS, XML, JSON) or a set of API endpoints that are required to access data.

[0246] With reference to FIG. 28, for a viewer client device wishing to view a replay video, an indication of available replay videos may be presented to the viewer client device for selection, or the viewer client device may request stream information using, for example, a title of the stream, one or more tags for the stream, or other criteria that may be used to identify an available replay video (e.g., any of the information stored in the database 420 that may be associated with a particular video available for replay). In particular, the viewer client device may send a request including one or more stream identifiers (e.g., a numerical and/or text identifier) for a previously recorded stream to the web server(s) 700 (via the API) to receive a copy of the recorded stream to serve as the replay video. In one example as discussed above, the web server(s) may first check the memory cache 460 for, or requests from the database 420, various stream information corresponding to the stream identifier(s) provided by the viewer client device. If the recorded stream does not exist, the web server(s) return an error message to the viewer client device. In other examples, reading from the database may or may not involve first checking a caching server for a cached response (and, if not found in the cache, then a direct database request is made).

[0247] If the recorded stream does exist, the web server(s) queries the database 420 to see if the recorded stream is associated with a particular event (or if instead it is associated more generally with a topic). If the stream is associated with an event, and the associated event is in the future (e.g., the recorded stream discusses predicted outcomes for a future event) or in progress, the web server(s) retrieve the event information and return to the viewer client device the stream information for the recorded stream and the event information for the associated event (e.g., an endpoint for the appropriate media source from which to obtain a copy of the recorded stream, and an endpoint for the appropriate socket of the socket server(s) dedicated to the event, and optionally initial event information). If, on the other hand, the recorded stream is not specifically associated with an event, or the event with which the recorded stream is associated is in the past, the process flow of FIG. 28 then calls the “Related Event Algorithm,” which is shown in FIG. 31 and discussed immediately below. Pursuant to the Related Event Algorithm, a current event (e.g., a currently-scheduled event) is selected that is in some manner related to the recorded stream serving as the replay video (whether the replay video itself is associated with a past event or, more generally, a topic of interest). Upon selection of the current event, event information for the current event is provided to the viewer client device in tandem with the replay video.

[0248] More specifically, FIG. 31 is a flow chart illustrating the process flow for a related event select algorithm (“Related Event Algorithm”) according to one example implementation, to facilitate the provision of a current event information endpoint (e.g., an appropriate socket of the socket server(s)) from which to provide the current event information and updates to same. In the example of FIG. 31, the replay video is considered to be generally related to the topic of sports, and in some instances the replay video may indeed be a recording of a live stream relating in some manner to a past or ongoing sporting event, or a future sporting event. Generally speaking, as noted above, the various stream information associated with the replay video and stored in the database 420 may include a title, one or more tags (e.g., including identifiers for sporting teams or participants, particular sporting events including dates and times, types of sport(s) or sporting event(s) to which the replay video pertain, leagues or other organizations relating to the sports topic of the replay video, etc.), and/or other information, and at least some of the stream information associated with the replay video may be used as part of the Related Event Algorithm. Examples of tags or other stream information that may be related to types of sporting events include, but are not limited to, “pre-season,” “exhibition,” “regular season,” “all-star/MVP,” “playoff,” and “championship.” Examples of tags or other stream information that may be related to teams or participants can include team names or participant names. Examples of tags or other stream information that may be related to a particular sporting league (i.e., a subcategorization of a sport) may include a league name, abbreviation or description, such as NFL, Euro Basketball, or Brazilian soccer. Examples of tags or other stream information that may be related to a particular type of sport include, but are not limited to, “baseball,” “basketball,” “football,” “soccer,” “hockey,” “tennis,” “golf,” etc.).

[0249] As shown in FIG. 31, if the stream information includes event information or otherwise indicates that the replay video is related to a particular event, the process selects the next event featuring either team or participant (e.g., selected in ascending chronological order) and, if such an event is found, the process returns event information relating to this next event. Otherwise, the process determines if the stream information includes team or participant information (e.g., a stream about the Ravens NFL team, or in the case of individual participants such as boxing, one of the fighters) and, if so, selects the next event featuring one or more of the teams or participants. Otherwise, if the stream information includes league information (for example, if the stream is about multiple NFL teams or if it is about a single NFL team and no upcoming event matched above), the process determines if there are any events for the league or related organization. Otherwise, if the stream information includes a sport designation (for example, if stream was about U.S. basketball but the U.S. basketball season is over), the process determines if there are other events for the same sport (e.g., in international markets). As a default, the process of FIG. 31 selects the next unstarted or in progress event for any team, participant, league, organization, sport or topic (e.g., as determined by the control server 500 via access to event information provider 55, such as STATS LLC). In one aspect, an eligible event refers to an event that has not ended, be it by expiring (time sensitive) or being closed (either by ending, or the data no longer updating).

[0250] Some variations are contemplated for the process shown in FIG. 31. For example, one or more teams or participants to which the replay video pertains may be given a “weighted preference,” i.e., specific teams or participants are given greater weight (e.g., either by the system or the generator of the recorded stream serving as the replay video) and events associated with teams/participants having a greater weight are checked for first at the appropriate stages of the process shown in FIG. 31. For example, if the Ravens team is given weighted preference, the process would check first if the Ravens have an event, and thereafter check the opponent. Another variation relates to “weighted preference in fallback,” e.g., checking for “top” leagues for upcoming events first before selecting the next possible event. Yet another variation relates to “viewer based weighting,” wherein event selection is based on weights provided by or generated for a particular viewer (e.g., based on geography). For example, if the viewer is an NFL fan (e.g., as indicated by profile information stored in database 420 and associated with their user account), the process of FIG. 31 may select NFL events if available and only if no NFL events are found then select any next event.

[0251] Regarding the various roles of the control server 500, and with reference again to FIG. 2, the control server 500 may communicate with multiple event information providers 55 and/or news feeds 65 to provide at least some of the event information in connection with a replay video. In such implementations, multiple instances of the service for Event Data Ingress (e.g., see FIG. 15) may be performed for respective providers of event information. In other aspects, an asynchronous task process service may be employed by the control server to retrieve information from multiple event information providers in parallel. In another aspect, respective instances of such a service may be customized to some extent based on the nature of the data and/or data format in which the event information provider makes the event information available. In general, as outlined in the example service shown in FIG. 15, the control server 500 retrieves data from a given event information provider, parses the data, normalizes the data, stores the data into the database, and pushes event information updates to appropriate socket channels.

[0252] FIGS. 32A, 32B and 32C illustrate a replay process similar to that shown in FIGS. 25A, 25B and 25C, to allow a viewer client device to replay a recorded video stream (“replay video”), according to one inventive implementation. Whereas the replay process shown in FIGS. 25A, 25B and 25C “replayed” the state of the event at the time of the original broadcast (per the process shown in FIGS. 26A, 26B and 26C), the alternate replay process now displays current (“real time” or “up to date”) event information that is in some manner related to the replay video, as discussed above. In particular, FIGS. 33A and 33B show a process flow illustrating an event data process called in the process of FIGS. 32A, 32B and 32C, according to one inventive implementation, to display current event information in tandem with a replay video. As shown in FIG. 32B, as playback of a replay video, or after a seek, at each “tick” (change of time index) the process branches to LOGIC-Process Replay Event Data outlined in FIGS. 33A and 33B.

[0253] Changing Events During Live Streams or Replays

[0254] In other inventive implementations, events may be changed during live streams and/or during replays. For live streams, a broadcaster or provider of the live stream can change the event during streaming. An example of a screen-shot of a display of a client device to facilitate changing events during streaming is shown in FIG. 34. In this implementation, a message is sent via the event socket and either contains the full information to generate an update or contains a command and required information (i.e, a new EventID) to have the client event information updated. In one aspect, the current event socket is disconnected and then connected to new event socket, or in another aspect a socket subscription to a data feed may be changed (e.g., via unsubscribe and subscribe). For replay streams, the viewer client device requests slices of the stream events that occurred during broadcast, and if the EventID changes, the viewer client device will be updated to connect to the new event information. On the Replay server side, the control server checks requested timestamp against logs and builds a response by running the Related Event Select Algorithm whenever there is an event change in the log. In one aspect, various commands in the sockets/replay events may be single action keywords or may include a keyword and then a data bundle containing information and properties to define subsequent actions.

[0255] Alternate Socket-Based Connections

[0256] With respect to sockets of the socket server (e.g., persistent http/2 connection), in one example implementation a viewer client device may connect to a socket server in multiple ways. In one example, the viewer client device makes one “master connection” to a socket server with multiple “subscriptions,” i.e., the viewer client device connects to the socket server and tells the socket server that it wants to subscribe to specific data feeds to receive information associated with a live stream or replay video, and in turn the socket server only pushes data pursuant to those subscriptions. In this manner, the viewer client may subscribe/unsubscribe at will (for a change of event for example). In another example, a given socket may be dedicated to providing information about multiple topics/events. In yet another example, as described elsewhere herein, each event may be associated with a corresponding socket and, as an event changes, connections to sockets are dropped and added as appropriate.

[0257] Topic Information

[0258] As noted above, in some implementations, a stream of digital content (e.g., video and/or audio) may relate to various topics of interest that may not have one or more temporal components; thus, such digital content may not be related to one or more events per se, but rather more generally may be related to one or more topics of interest (e.g., a classification of subject matter that may or may not include a temporal component). Accordingly, all events include one or more topics of interest, but not all topics of interest are associated with events. Examples of various topics of interest include, but are not limited to, healthy eating or dieting, cooking or baking, gardening, religion, dating, politics, culture, art, music, travel, history, architecture, animal life, education and instruction, playing a musical instrument, learning a language, auto repair, real estate, business, economics (e.g., finance and world markets), legal issues, global warming, space exploration, a particular TV program or series, a particular entertainment or sports personality, video games, and a wide variety of hobbies.

[0259] With reference again to FIG. 20A (see the option “Stream Type?—Anything”), upon generation of a stream of digital content the generator of the stream (e.g., a broadcaster) may provide a title for the stream as well as one or more tags to be associated with the stream; such tags may facilitate searching and various social media functionality (e.g., to allow prospective viewers to search for and find the stream based on various criteria represented by the tags). The stream title and one or more tags for the stream are all stored in the memory system 400 (e.g., the database 420) shown in FIG. 2. The title and/or one or more tags associated with the stream may indicate one or more topics associated with the stream. Alternatively, while not shown explicitly in FIG. 20A, various topics may be previously stored in the memory system 400, and such previously-stored topics may be presented as a list for selection of one or more topics to be associated with a stream of digital content.

[0260] In view of the foregoing, yet other implementations of the inventive concepts herein relate to providing topic information in tandem with a stream of digital content whose subject matter relates in some manner to one or more particular topics. To this end, FIG. 35 provides a block diagram similar to that shown in FIG. 2 for the servers and memory storage devices 1000 in which the control server 500 may be coupled to a topic information provider 57 (e.g., in a manner similar to that discussed above for the event information provider 55 and the news feeds 65) to retrieve various topic information. In this implementation, a given socket of the socket server(s) 600 may be dedicated to providing such topic information in a manner similar to that described herein for event information. Examples of a topic information provider 57 include, but are not limited to, various Internet information sources such as Wikipedia (en.wikipedia.org/wiki/Main Page), the Yahoo Finance API (for real time market information), the Coinmarketcap API (for real time cryptocurrency pricing), and Ticketmaster (ticketmaster.com; for real time ticketing availability). Similarly, one or more of the news feeds 65 also (or alternatively) may serve as a topic information provider (e.g., to determine when new articles are published). More generally, a topic information provider 57 may be any one of a number of Internet information sources that provide real-time information on a given topic.

[0261] With reference to FIG. 35, in one example implementation one or more topic information providers 57 may be selected by the control server based at least in part on a title, one or more tags, or other stream information stored in the memory system 400 in connection with the stream of digital content. In one aspect, the control server 500 periodically retrieves various topic information from the topic information provider 57 that is germane to a stream of audio and/or video digital content. In another aspect, the control system 500 may store at least some portion of retrieved topic information in the memory system 400. The control server may then pass first topic information 507A and the second topic information 507B to the socket server(s) 600. In turn, the socket server(s) 600 establish one or more first topic sockets 607A dedicated to the first topic information and one or more second topic sockets 607B dedicated to the second topic information.

[0262] In FIG. 35, in one aspect the web server(s) 700 may provide to the first viewer client device 200A a first topic identifier (a first TopicID) that corresponds to the first topic socket(s) 607A; the web server(s) 700 also may provide to the second viewer client device 200C a second topic identifier (a second TopicID) that corresponds to the second topic socket(s) 607B. The first viewer client device 200A uses the first TopicID to connect to the first topic socket(s) 607A (e.g., via a first URL including the first TopicID in a path of the URL), and the second viewer client device 200C uses the second TopicID to connect to the second topic socket(s) 607B (e.g., via a second URL including the second TopicID in a path of the URL). The first topic information 507A is then transmitted to the first viewer client device 200A via a first topic information Internet communication channel 207A between the first topic socket(s) 607A and the first viewer client device 200A, and the second topic information 507B is transmitted to the second viewer client device 200C via a second topic information Internet communication channel 207C between the second topic socket(s) 607B and the second viewer client device 200C.

[0263] With reference again to FIG. 21A, in one example implementation a “stream active” process similar to 2102A shown in FIG. 21A would have an additional (or alternative) step that queries if the active stream is related to a topic (e.g., is the active stream associated with a TopicID) and, if so, the process would connect to one or more particular sockets of the socket server(s) 600 for the topic information communication channel based on the TopicID, for example. In a manner similar to that of event information communication channels, in one aspect a given topic information communication channel is established by a persistent (and, in some cases, authenticated) connection between a given viewer client device and a given socket of the socket server (such that topic information and updates to same is/are pushed from the socket to the viewer client device).

[0264] In the implementation shown in FIG. 35, another significant role of the control server 500 thus relates to the collection of topic information (e.g., from external Internet providers), maintaining relevant topic information in the database 420, and distributing the collected topic information to multiple broadcaster and/or viewer client devices in a relatively low-latency and synchronized manner (e.g., with respect to a stream of digital audio/video content).

[0265] As noted above, the technical challenge of displaying topic information and any updates to same in a synchronized and low-latency manner amongst multiple client devices is addressed in part by using a single control server 500 to gather and parse topic information. This single point of entry of topic information prevents synchronization errors inherent in network communications. Once a change in topic information has been detected (e.g., via periodic queries of the topic information provider 57), the control server provides these changes to the one or more sockets dedicated to the topic (to which multiple client devices are communicatively coupled), resulting in a single synchronized update to all client devices and thereby significantly mitigating client-by-client latency and/or synchronization issues.

[0266] In some example implementations, the control server 500 implements two service methods relating to topic information, namely, a topic data ingress service and a topic data monitor service. The topic data ingress service is performed with a first periodicity (e.g., once or twice a day) to maintain and update a topic list in the database 420. The topic data monitor service is performed with a second and more frequent periodicity (e.g., once a minute) to check for any topics for which updated information may be available (e.g., new stock prices, new articles being published, updated availability of tickets for an upcoming concert or game) and, if found, to retrieve fresh data about topic updates from the topic information provider (e.g., at an even greater frequency, for example once a second).

[0267] FIG. 36 shows a process flow diagram illustrating an topic data ingress service method 3602 performed by the control server 500, according to one inventive implementation. In some examples, the control server also may perform a topic data monitor service method, in tandem with a topic data ingress service, to query the topic information provider for a list of all topics that may be particularly trending in a specified window around the current time (e.g., a 48 hour window), to allow tracking of trending topics. For each trending topic, a topic clock and other topic information are updated frequently (e.g., once a second) to provide regular updates of topic information messages that are broadcast to one or more dedicated topic information sockets of the socket server(s) 600.

[0268] With reference to FIG. 36, a topic information provider is contemplated as supporting multiple different topics for furnishing topic information. For each topic, the control server retrieves the raw information provided by the topic information provider, and in some instances converts and/or compresses the raw information to provide a standardized format of essential data elements for storing in the database 420 and/or distribution to client devices (e.g., via broadcast of topic messages having the standardized format to one or more dedicated sockets of the socket server(s) 600). Examples of data elements for topic information include, but are not limited to, a text identifier for the topic, a numeric identifier for the topic (e.g., TopicID), one or more tags associated with the topic, quantized information germane to the topic, identifiers for particular personalities or things associated with or related to the topic, a date and/or time of new/updated information published about the topic, associated topics, events associated with topics, and latest news relating to a topic. In some examples, the control server further normalizes the date and/or time to a particular reference frame (e.g., converting from UTC to EST/EDT). In addition, all incoming data elements may or may not be further normalized to standardize the provided information from multiple topic information providers to facilitate consistent formatting and accessibility of data in the database 420.

[0269] Digital Content Streaming with Online Gaming Information

[0270] As noted above, one example of event information associated with a stream of digital content associated with an event includes online gaming information. FIG. 37 illustrates a screen shot of a display 250-1 of an example viewer client device, showing various displayed content including a live stream 252-1 of a sporting event (e.g., a football game between the Baltimore Ravens and the Dallas Cowboys). In the example implementation illustrated in FIG. 37, the live stream 252-1 of the sporting event is displayed together with event information 254-1 regarding the sporting event, wherein the event information may include not only score information 254-1A but also online gaming information 254-1B, for example, relating to one or more bets that may be made in connection with the sporting event. While not explicitly shown in FIG. 37, it should be appreciated that, as discussed above, chat messages from one or more viewers also may be displayed in the display 250-1; additionally, various graphics, special effects and/or animation elements 256-1 may be included in the content rendered on the display 250-1 (e.g., as one or more overlays, some of which are rendered in a “lower third” of the display 250-1). In connection with the display shown in FIG. 37, it should be appreciated that, in other implementations, the event information displayed may include only the online gaming information 254-1B, and need not necessarily include score information 254-1A.

[0271] More specifically, as shown in FIG. 37, the online gaming information 254-1B may include one or more display widgets that render content relating to one or more bets that may be made in connection with the sporting event, including more general bets (e.g., “spread,” “line,” “over/under”) as well as one or more specific bets relating to various occurrences, milestones, target statistics, etc., during the game (e.g., “last touchdown,” “total touchdowns”). As discussed above in connection with the display of event information more generally, all of the online gaming information 254-1B, or one or more parts of the online gaming information 254-1B, may be interactive (e.g., the viewer may press or thumb-over one or more portions of the online gaming information to launch further graphics or animations, receive additional online gaming information and/or additional information about the sporting event, or navigate to another Internet location to receive additional online gaming information and/or additional information relating to the live sporting event).

[0272] Regarding the displayed video content 252-1 in the example of FIG. 37, in other example implementations the displayed video content 252-1 may not be a live stream but instead may be a replay video, as discussed above in connection with FIGS. 28-33. More specifically, the displayed video content 252-1 may be a replay of a recording of a live stream relating to a past event or an ongoing event that has already started. In this example, the servers and memory storage devices 1000 may provide to the viewer client device, in tandem with the replay video, current (real-time or essentially real-time) online gaming information 254-1B that is in some manner related to one or more aspects of the replay video.

[0273] In the inventive implementation shown in FIG. 37, and with reference again to FIG. 35, one or more event information providers 55 from which the control server 500 ingresses various event information 502A, 502B, may provide a variety of online gaming information for display in connection with streaming digital content. Examples of event information providers 55 that may provide various types of online gaming information relating to sporting events includes, but is not limited to, Panda, FanDuel SportsBook, Bovada, MGM SportsBook, and William Hill.

[0274] With respect to the ingress by the control server 500 of online gaming information, in a manner similar to other implementations discussed herein the control server collects online gaming information (e.g., from external Internet providers), maintains relevant online gaming information in the database 420, and distributes the collected online gaming information to multiple client devices in a relatively low-latency and synchronized manner (e.g., with respect to the live stream of the sporting event).

[0275] In some implementations, the technical challenge of displaying online gaming information and updates to same in a synchronized and low-latency manner amongst multiple client devices is addressed in part by using a single control server 500 to gather and parse online gaming information captured in real-time. For example, for sporting events, online gaming information may be obtained by the single control server from one or more dedicated third-party providers (e.g., WilliamHill and FanDuel SportsBook). This single point of entry of online gaming information prevents synchronization errors inherent in network communications. Once a change in online gaming information has been detected (e.g., the betting odds have changed), the control server provides these changes to the one or more sockets dedicated to the event (to which all viewers and broadcasters of video-based commentary regarding the event are communicatively coupled), resulting in a single synchronized update to all client devices and thereby significantly mitigating client-by-client latency and/or synchronization issues.

[0276] In some example implementations, the control server 500 may implement two service methods relating to online gaming information, namely, an online gaming data ingress service (which may include multiple parallel or sequential services, e.g., for respective online gaming information providers) and a priority online gaming data monitor service. The online gaming data ingress service may be performed with a first periodicity (e.g., every 5 minutes for overall betting information) to maintain and update the online gaming information and associated event list in the database 420. The priority online gaming data monitor service is performed with a second and more frequent periodicity (e.g., once a minute) to check for any events that are either in progress or are given priority due to their usage in a currently live broadcast or a broadcast being replayed and, if found, to retrieve current online gaming information associated with the events from the online gaming information provider(s).

[0277] FIG. 38 shows a process flow diagram illustrating an online gaming data ingress service method 3802 performed by the control server 500, according to one inventive implementation, and FIG. 39 show a process flow diagram illustrating a priority online gaming data monitor service method 3902 performed by the control server according to one inventive implementation. In these methods, an online gaming information provider is contemplated as supporting multiple different types of events for furnishing information (various types of sporting events such as basketball, football, baseball, hockey, etc.), and providing information for each instance of an event of a given event type (e.g., information for each of multiple basketball games, each of multiple football games, each of multiple baseball games).

[0278] For each event, the control server retrieves the raw information relating to a given event (as discussed above in connection with FIG. 15), and also retrieves information provided by the online gaming information provider, and in some instances converts and/or compresses the raw information to provide a standardized format of essential data elements for storing in the database 420 and/or distribution to client devices (e.g., via broadcast of event messages having the standardized format to one or more dedicated sockets of the socket server(s) 600). Examples of data elements for event information were discussed above in connection with FIG. 15; in FIG. 38, examples of data elements for online gaming information include currently available bets and odds data, which may include, but is not limited to, money line, spread, over/under, outright winner, draw, win-place-show, parlay options, period outcome bets, and various event-specific proposition bets (e.g., player X will score 10 or more points). Available bets and odds data are event specific and category based and, as such, may vary depending on the event and the type of event (e.g., win or draw for soccer games).

CONCLUSION

[0279] While various inventive implementations have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein, and each of such variations and/or modifications is deemed to be within the scope of the inventive implementations described herein. More generally, those skilled in the art will readily appreciate that all parameters and configurations described herein are meant to be exemplary inventive features and that other equivalents to the specific inventive implementations described herein may be realized. It is, therefore, to be understood that the foregoing implementations are presented by way of example and that, within the scope of the appended claims and equivalents thereto, inventive implementations may be practiced otherwise than as specifically described and claimed. Inventive implementations of the present disclosure are directed to each individual feature, system, article, and/or method described herein. In addition, any combination of two or more such features, systems, articles, and/or methods, if such features, systems, articles, and/or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0280] The above-described implementations can be implemented in multiple ways. For example, implementations may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Further, it should be appreciated that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.

[0281] Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format. Such computers may be interconnected by one or more networks such as Internet. The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0282] In this respect, various inventive concepts may be embodied as a computer readable memory or storage medium (or multiple computer readable storage media) (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other non-transitory medium or tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various implementations of the invention discussed above. The computer readable medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present invention as discussed above.

[0283] Unless otherwise indicated, the terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of implementations as discussed above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present invention.

[0284] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various implementations.

[0285] Also, data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements. In some implementations, a schema-minimal storage system may be implemented in a relational database environment using key-value storage versus defined data structures.

[0286] With the foregoing in mind, each of the client devices described herein, as well as various servers and other computing devices of the broadcast/viewing servers and memory storage devices shown for example in FIGS. 2 and 3, may comprise one or more processors, one or more memory devices or systems communicatively coupled to the one or more processors (e.g., to store software code and other data), and one or more communication interfaces communicatively coupled to the one or more processors so as to implement the various specific and inventive functionality described herein.

[0287] Also, various inventive concepts may be embodied as one or more methods, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, implementations may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative implementations.

[0288] All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety.

[0289] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and/or ordinary meanings of the defined terms.

[0290] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0291] The phrase “and/or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and/or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and/or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one implementation, to A only (optionally including elements other than B); in another implementation, to B only (optionally including elements other than A); in yet another implementation, to both A and B (optionally including other elements); etc.

[0292] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items in a list, “or” or “and/or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0293] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and/or B”) can refer, in one implementation, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another implementation, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another implementation, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0294] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of” and “consisting essentially of” shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.