Universal Mirroring Receiver

Abstract

This invention describes software which allows a single content receiver to receive minoring streams from multiple heterogeneous platforms

Claims

1-7. (canceled)

8. A receiver device comprising: a memory storing instructions that, when executed by the receiver device, cause the receiver device to: receive a first mirroring stream from a first device and a second mirroring stream from a second device, the first and second mirroring streams conforming to different vendor-provided screen mirroring protocols; identify, by a first protocol handler, a screen mirroring protocol of the first mirroring stream, the first mirroring stream comprising a first encoded video data and a first encoded audio data; decode, by a first video decoder, the first encoded video data to generate a first decoded video data; decode, by a first audio decoder, the first encoded audio data to generate a first decoded audio data; identify, by a second protocol handler different from the first protocol handler, a screen mirroring protocol of the second mirroring stream, the second mirroring stream comprising a second encoded video data; decode, by a second video decoder, the second encoded video data to generate a second decoded video data; output, by an audio renderer, the first decoded audio data; receive, by a video layout manager, the first decoded video data and the second decoded video data; produce, by the video layout manager, a single stream comprising the first decoded video data and the second decoded video data, wherein the single stream is position-optimized and scaled for displaying a combined final image on a viewing device; and transmit the single stream to the viewing device.

9. The receiver device of claim 8, wherein the first encoded video data and the first encoded audio data are multiplexed.

10. The receiver device of claim 8, wherein rotation of a first mirrored content, represented by the first decoded video data, in a display area of the viewing device does not cause repositioning of a second mirrored content, represented by the second decoded video data, in the display area of the viewing device.

11. The receiver device of claim 10, Wherein at least one of the first decoded video data and the second decoded video data in the single stream is configured to have a 1:1 aspect ratio.

12. The receiver device of claim 8, wherein the video layout manager determines whether to simultaneously display a mirrored content, represented by the first decoded video data, and the second mirrored content, represented by the second decoded video data, on the viewing device in a single row.

13. The receiver device of claim 8, wherein the memory stores further instructions that, when executed by the receiver device, cause the video layout manager to receive a first orientation information associated with the first decoded video data and a second orientation information associated with the second decoded video data.

14. The receiver device of claim 8, wherein the vendor-provided screen mirroring protocols are selected from a group comprising an AirPlay protocol, a Miracast protocol, and a Google Cast protocol.

15. A method of simultaneously mirroring heterogeneous mirroring streams, the method comprising the steps of: receiving a first mirroring stream from a first device and a second mirroring stream from a second device, the first and second mirroring streams conforming to different vendor-provided screen mirroring protocols; identifying, by a first protocol handler, a screen mirroring protocol of the first mirroring stream, the first mirroring e comprising a first encoded video data and a first encoded audio data; decoding, by the first video decoder, the first encoded video data to generate a first decoded video data; decoding, by the first audio decoder, the first audio data to generate a first decoded audio data; identifying, by a second protocol handler different from the first protocol handler, a screen mirroring protocol of the second mirroring stream, the second mirroring stream comprising a second encoded video data; decoding, by the second video decoder, the second encoded video data to generate a second decoded video data; outputting, by an audio renderer, the first decoded audio data and the second decoded audio data; receiving, by a video layout manager, a decoded first video data; producing, by the video layout manager, a single stream comprising the first decoded video data and the second decoded video data, wherein the single stream is position-optimized and scaled for displaying a combined final image on a viewing device; and transmitting the single stream to the viewing device.

16. The method of claim 15, wherein, in the first mirroring stream, the first encoded video data and the first encoded audio data are multiplexed.

17. The method of claim 15, wherein rotation of a first mirrored content, represented by the first decoded video data, in a display area of the viewing device does not cause repositioning of a second mirrored content, represented by the second decoded video data, in the display area of the viewing device.

18. The method of claim 17, wherein at least one of the first decoded video data and the second decoded video data in the single stream is configured to have a 1:1 aspect ratio.

19. The method of claim 15, wherein the video layout manager determines whether to simultaneously display a first mirrored content, represented by the first decoded video data, and a second mirrored content, represented by the second decoded video data, on the viewing device in a single row.

20. The method of claim 15, further comprises the step of receiving, by the video layout manager, a first orientation information associated with the first decoded video data and a second orientation information associated with the second decoded video data.

21. The method of claim 15, wherein the vendor-provided screen mirroring protocols are selected from a group comprising an AirPlay protocol, a Miracast protocol, and a Google Cast protocol.

22. A non-transitory computer readable medium having stored thereon instructions that when executed by a receiver device, cause the receiver device to: receive a first mirroring stream from a first device and a second mirroring stream from a second device, the first and second mirroring streams conforming to different vendor-provided screen mirroring protocols; identify, by a first protocol handler, a screen mirroring protocol of the first mirroring stream the first mirroring stream comprising a first encoded video data and a first encoded audio data; decode, by the first video decoder, the first encoded video data to generate a first decoded video data; decode, by the first audio decoder, the first encoded audio data to generate a first decoded audio data; identify, by a second protocol handler different from the first protocol handler, a screen mirroring protocol of the second mirroring stream, the second mirroring stream comprising a second encoded video data; decode, by the second video decoder, the second encoded video data to generate a second decoded video data; output, by an audio renderer, the first decoded audio data; receive, by a video layout manager, the first decoded video data and the second decoded video data; produce, by said video layout manager, a single stream comprising the first decoded video data and the second decoded second video data, wherein the single stream is position-optimized and scaled for displaying a final image on a viewing device; and transmit the single stream to the viewing device.

23. The non-transitory computer readable medium of claim 22, wherein the first encoded video data and the first encoded audio data are multiplexed.

24. The non-transitory computer readable medium of claim 22, wherein rotation of a first mirrored content, represented by a first decoded video data, in a display area of the viewing device does not cause repositioning of a second mirrored content, represented by the second decoded video data, in the display area of the viewing device.

25. The non-transitory computer readable medium of claim 24, wherein at least one of the first decoded video data and the second decoded video data in the single stream is configured to have a 1:1 aspect ratio.

26. The non-transitory computer readable medium of claim 22, wherein the video layout manager determines whether to simultaneously display a first mirrored content, represented by the first decoded video data, and a second mirrored content, represented by the first decoded video data, on the viewing device in a single row.

27. The non-transitory computer readable medium of claim 22, wherein the stored instructions, when executed by the device, cause said video layout manager to receive a first orientation information associated with the first decoded video data and a second orientation information associated with the second decoded video data.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 is an illustration of the present invention, which receives input from a first and second protocol signal;

[0015] FIG. 2 illustrates the receiver receiving signals via multiple protocols and combining them into a single video and audio stream to be rendered by a receiver's audio and video renderers; and

[0016] FIG. 3 illustrates the simultaneous multiple video handling of the receiver.

DETAILED DESCRIPTION OF THE INVENTION

[0017] Certain terminology is used herein for convenience only and is not to be taken as a limitation on the present invention. The terminology includes the words specifically mentioned, derivatives thereof and words of similar import. The embodiments discussed herein arc not intended to he exhaustive or to limit the invention to the precise form disclosed. These embodiments arc chosen and described to best explain the principle of the invention and its application and practical use, and to enable others skilled in the an to best utilize the invention.

[0018] Referring to FIG. 1, in a first embodiment the invention is directed to a device of the present invention illustrated in relation to the incoming signals and displays. As illustrated, the receiver 10 provides the transformation of the two signals, as illustrated for example purposes, one front an AirPlay capable device and one from a Miracast capable device, for presentation on any output device receiving the “mirrored platforms”. However, one skilled in the art would appreciate that the receiver 10 could handle multiple signals in an amount greater than two.

[0019] Referring to FIGS 2 and 3, in a first embodiment, the invention is directed to a receiver 10 for heterogeneous platforms including a first protocol handler 12A and a second protocol handler 12B; us illustrated in the present embodiment “AirPlay Protocol Handler” and “Miracast Protocol Handler” respectively. The inclusion of additional heterogeneous platforms would require additional protocol handlers. A separate protocol handler for each new protocol is required as each protocol can differ significantly, for example in how it transports audio and video. Some protocols transport it “joined” or “multiplexed” using the same transport mechanism (as in the case of Miracast) while others (such as AirPlay) use separate transport mechanisms for audio and video. The protocol handlers 12A and I2B include unique software code that identifies what protocol is being transmitted and selects the appropriate conversation mode, similar to a person hearing a language and switching to that language in order to converse. The code supports “conversation” with the remote device via the protocol. One skilled in the art will appreciate that the protocols (AirPlay and Miracast) are “spoken languages”; each protocol is a type of a “spoken language”. Much like a language is native to a person “spoken language”; a person can learn other languages (non-native) and can therefore speak more than one language. A “spoken language” is a protocol, both “native and non-native”; for illustration Airplay is from Apple™ and Miracast is front Wi-Fi Alliance™.

[0020] In the present invention, the software has the ability to 1) recognize the language (protocol) of each incoming signal, and 2) adjust (based on the protocol) to provide the necessary sequential steps to support the resultant “mirroring” from multiple sending protocols. The core of this invention is to mirror signal front multiple senders. However, one skilled in the art would recognize there is nothing that would prevent the device from directing its output to multiple recipients. The concept of the present invention includes mirroring to multiple recipients as appreciated by the scope of the information in this document.

[0021] The first protocol handler 12A emits a video stream to a first protocol handler video decoder 14A, and an audio stream to an audio decoder 15A connected to the first protocol handler 12A's output, “Audio+time sync”. Therefore, the audio and video have been separated and have associated time synchronization information. The time synchronization is then used by an internal clock of the computing device running the present invention, which ensures that the video and audio is correctly synchronized when it is processed by the video and audio renderers (e.g. displayed on a screen or played in speakers). As illustrated, both audio streams go through the audio renderer, as would all other audio from additional protocols if included.

[0022] The second protocol handler 12B emits a multiplexed signal to a demultiplexer 18. The multiplexed signal contains potentially multiple streams, normally an audio stream and a video stream. A “demultiplexer” is a module that separates an input signal into separate data streams. A demultiplexer selects one of several analog or digital input signals and forwards the selected input signal into a single line. Each “parcel” of data, being either a multiplexed audio or video frame within the multiplexed signal, will have an identifier specifying which of these streams it belongs to. An “Audio decoder” module 15B connected to the demultiplexer 18 “output” labeled “Audio time sync” is illustrated. As illustrated, the paths of the signal for AirPlay and Miracast are different because each protocol can transport audio and video either through a single transport mechanism (e.g. Miracast) or use separate transport mechanism for audio and video (e.g. Airplay). In the case of a single transport mechanism, a demultiplexer is needed to separate the audio and the video into two streams.

[0023] The demultiplexer 18 emits a video stream to a second protocol handler video decoder 14B and an audio stream to the audio decoder 15B which is then sent to the audio Tenderer 16. Thu first protocol handler video decoder 14A and second protocol handler video decoder I4B each submit a stream to a video layout manager 20.

[0024] Referring to FIG. 3, the video layout manager 20 receives two signals (each signal containing video frames) but emits only one. As a unit, the video layout manager 20 integrates multiple components including an image positioning module 112 which receives a video stream from the first protocol handler video decoder 14A, and a video stream from the second protocol handler video decoder 14B. A scaling module 114 and an image enhancement module 116 receive the single emitted stream from the image positioning module 112.

[0025] The receiver of the present invention recognizes the display area of a “viewing device”, e.g. TV or monitor, etc., that is available to display the combined (single stream) final image. The image positioning module 112 arranges the layout of the input video signals so that they use the available space (the horizontal and vertical resolution of the output video signal) efficiently. In the case of two devices, on a normal monitor with a 16:9 aspect ratio, one video signal will be rendered on the left and the other on the right. If a third device joins, either each device will be displayed on the same row or a new row might be added with two signals being displayed in the first row (one to the left, the other to the right) and the third signal displayed in the second row. The algorithm of the software regards the input signal as having a square resolution (using the greater of width or height) so as to allow the device to rotate on screen freely, without causing other device display areas to move in the final output.

[0026] As best illustrated in FIG. 3, the single emitted stream front the image positioning module 112 to the scaling module 114 has “position optimized full resolution image frames” and is emitted thereafter to the image enhancement module 116. Collectively, the image positioning module 112, scaling module 114 and image enhancement module 116 are referred to as the “video layout manager”, best illustrated in FIG. 3. The protocol handlers 12A, 12B inform the image positioning module 112 when a new connection has been made by signaling it and passing it the resolution of the new video stream. The protocol handlers 12A, 12B also inform the image positioning module 112 when a device is disconnected. These connect and disconnect events cause a new layout to be made for positioning the incoming video streams in the final display.

[0027] Referring again to FIG. 2, the single emitted stream from the video display manager 20 is directed to the video render 22.

[0028] As discussed, one skilled in the art would recognize the present embodiment illustrates two incoming connections, but could include multiple incoming connections, allowing the receiver to act as a conduit for the streams of an unlimited number of devices.

[0029] The invention will now be described in additional embodiments wherein like reference numbers are meant to illustrate like features.

[0030] As best illustrated in FIG. 2, in another embodiment, the invention is directed to a method for combining “transmitted data”/“network packets” provided by platform manufacturers into a single receiver accepting from heterogeneous platforms. The method including the sequential steps of preparing the receiver of the previous embodiment. After preparation of the receiver, the method continues by receiving at least two video signals from the heterogeneous platforms by the receiver and receiving at least two audio signals from the heterogeneous platforms. The at least two video signals from heterogeneous platforms are synchronized with their audio signals in the receiver, wherein the synchronized signals arc directed from at least two video signals and audio signals from heterogeneous platforms in the receiver to a video renderer and a sound renderer. The method concludes by simultaneously distributing the synchronized video signals and audio signals to available renderers (e.g. display, connected projector, speakers, and headphones).

[0031] In yet another embodiment. the invention is directed to computer-readable medium storing software that combines “transmitted data”/“network packets” provided by platform manufacturers into a single receiver accepting from heterogeneous platforms. The software includes executable code that receives at least two video signals from the heterogeneous platforms by the receiver and receives at least two audio signals from the heterogeneous platforms. The executable code synchronizes the video signals and audio signals from heterogeneous platforms in the receiver and directs synchronized signals from the at least two video signals from heterogeneous platforms in the receiver to a video renderer (such as a video card). The executable code simultaneously distributes the synchronized signals from at least two audio signals to a sound renderer (such as a sound card).

EXAMPLES

[0032] As discussed herein and best illustrated in FIGS. 1, 2 and 3, in a non-limiting example, the receiver 10 provides the transformation of the two signals, one from AirPlay and one from Miracast, for presentation on any device 10 receive the “mirrored platforms”. Within the receiver 10, the method has multiple steps directed by software code executed on a computing device such as a PC, Mac, computing stick or any platform capable of executing computer code to perform this transformation. The present invention provides a receiver that is capable of accepting mirroring from any type of device independent of its type. Thus, as illustrated in FIGS. 1, 2 and 3 the present invention allows a Miracast capable sending device and an AirPlay capable sending device, to mirror to an external display at the same time. The user of the receiver is not restricted to mirroring from any particular type of device and therefore, a user can bring their personal device to a meeting, school or any situation where content needs to be shared. The video output signals of the present invention are directed to the “monitor” selected in the software's settings. Typically, this is a physical device such as a monitor or a projector connected to the computer running the software. The audio signals are directed to the “output” device selected in the software's settings. Typically, this is the speaker output on a sound card.

[0033] The present invention provides the capability to interact with multiple types of devices. More particularly, a user with an iOS device, another with an Android device and yet another one with a Windows device can all at the same lime mirror their content to the external display using the present invention; commercially named AirServer Universal™.

[0034] Another example of the present invention providing a beneficial situation would be a classroom setting. A teacher has a device for sharing information with students via mirroring. Students have their own devices and can share their screen (which may, for example, contain solutions to school assignments) by mirroring at the same time. Without AirServer Universal™ they would all need to decide to use one type of device to achieve this. With AirServer Universal™, students can bring their device of choice (iOS, Windows, Android based devices) and have no restrictions on who can mirror to the external display.

[0035] An additional example can be directed to the field of healthcare, wherein two health care professionals comparing notes on a patient, like x-rays having two different types of devices can now compare these side by side on a large screen. Further one skilled in the art would appreciate the present invention would allow the ability to “overlay” multiple types images, e.g. x-rays, for comparison purposes. More particularly, the present invention would allow the multiple images to be “overlaid” so as to view progression of a malady by size or time. e.g. progression of tumor size or increase in amount of cells.

[0036] Yet another application of the invention is in the field of gaming, wherein two or more players of a computer game can simultaneously project their gaming sessions from their devices for the purpose of comparison or entertaining an audience.

[0037] A further example can be found in the field of software development, wherein two or more software developers/designers interact to take a reference design of user interaction and ensure the design is working properly or as designed by mirroring the reference design of one device next to an implementation of the same software on another platform. This is performed in software quality assurance testing using tire present invention.

[0038] It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept thereof. It is understood, therefore, that this invention is not limited to the particular embodiments disclosed, but it is intended to cover modifications within the spirit and scope of the present invention as defined by the appended claims.