METHOD FOR STREAMING AND REPRODUCING APPLICATIONS VIA A PARTICULAR TELECOMMUNICATIONS SYSTEM, TELECOMMUNICATIONS NETWORK FOR CARRYING OUT THE METHOD, AND USE OF A TELECOMMUNICATIONS NETWORK OF THIS TYPE
20190253753 · 2019-08-15
Inventors
Cpc classification
H04N21/23424
ELECTRICITY
H04N21/4351
ELECTRICITY
H04N21/2387
ELECTRICITY
H04N21/4384
ELECTRICITY
International classification
H04N21/435
ELECTRICITY
H04N21/234
ELECTRICITY
H04N21/438
ELECTRICITY
H04N21/235
ELECTRICITY
Abstract
The invention relates to a method for streaming and reproducing applications (apps) via a particular telecommunications system. The invention also relates to a telecommunications network. The invention further relates to the use of a telecommunications network of this type for streaming and reproducing applications (apps) via a particular telecommunications system. The invention furthermore relates to an application which makes it possible to configure the download, installation and updating of data on a terminal such that the whole application is fully executable on the terminal while downloading, installing and updating takes place on the terminal.
Claims
1. A method for streaming and reproducing applications (APPs) via a particular telecommunications system, in which one or more streaming servers, which can be connected together by telecommunication, execute the relevant application and which form a connection to the respective telecommunication terminal which is characterized by an optimal connection, taking account of the number of hops, latency, bandwidth and ISP, wherein the relevant telecommunication terminal retrieves the desired application from a streaming server with optimal connection, which provides the processing power for setting up the video stream and the encoding of the relevant application, for the reproduction of applications on non-native application system environments, which differ ether due to different hardware or software components, wherein the streaming server undertakes the handling of the different applications and the rendering/encoding of the application and its audio and video signals, wherein the data are transmitted to the respective telecommunication terminal mobile wireless device, tablet, laptop, PC, TVand the transmission is performed using a modified h.264 protocol and the WAN is used as the transmission medium for audio/video packets via UDP, TCP and the entire processing capacity is used by the relevant streaming server, wherein the packet-based data are decoded only at the telecommunication terminal, wherein in order to provide a platform-independent streaming technology, which is programmed once and is portable to any telecommunication terminals, in which the streaming of individual applications, for example video games, is carried out over a WAN in such a way that a) a communication with the session server is performed by means of the telecommunication terminal (small applications); b) a specific session is implemented for a specific end-user for the streaming server of the relevant application, for example a game, which is optimally nearest to the telecommunication terminal; c) session information is communicated to the telecommunication terminal and the streaming server by the relevant session server; d) a direct connection is implemented between the telecommunication terminal and the streaming server of the relevant application, for example a video game; e) when setting up a direct connection between the telecommunication terminal and the relevant streaming server, the following steps are initiated: i. recording the audio/video data of the running application, for example a game, via the streaming server on which the game is running; ii. compressing the audio/video data using high-quality run-time modified hardware encoders; iii. transmission of the compressed audio/video data via WAN; iv. receiving the audio/video data by the telecommunication terminal; v. decompressing the audio/video data using a modified decoder; vi. reproduction of the audio/video data on the telecommunication terminal (small); vii. recording the actions (inputs) of the user of the telecommunication terminal, for example a player, on the telecommunication terminal (small) by a Thin Layer, which is placed dynamically over the display field; viii. efficient transmission of the inputs back to the relevant streaming server of the game and ix. playback of the transmitted inputs on the streaming server, wherein in the event of packet loss during the transmission of files to the telecommunication terminal, for example from a streaming server to the telecommunication terminal, making use of a plurality of specially developed error correction procedures, the following steps are carried out: a) recovery strategy is invoked on telecommunication terminal (small) in order to maintain a smooth gaming experience; b) the appropriate recovery strategy is selected, and c) the recovery request is sent back to the relevant streaming server of the application, for example the game, this process can also be initialized by the streaming server, wherein the applications are streamed on one or more local terminals independently of the downloading on the local terminal, and that for the method a source codefor communication with a client (user, terminalis used as follows: TABLE-US-00003 {circumflex over ()}*********************** AddPortAsynchronisation.java ********************************* ********************************************************************************************** / package org.cloundgaming4u.client.portforwarding; import java.io.lOException; import net.sbbi.upnp.messages.UPNPResponseException; import android.content.Context; import android.os.AsyncTask; import android.util.Log; public class AddPortAsync extends AsyncTask<Void, Void, Void> { private Context context; private UPnPPortMapper uPnPPortMapper; private String externallP; private String internallP; private int externalPort; private int internalPort; public AddPortAsync(Context context, UPnPPortMapper uPnPPortMapper, String externallP, String internallP, int externalPort, int internalPort) { this. context = context; this. uPnPPortMapper = uPnPPortMapper; this. externallP = externallP; this. internallP = internallP; this. externalPort = externalPort; this. internalPort = internalPort; } @Override protected void onPreExecute( ) { super. onPreExecute( ); if(uPnPPortMapper == null) uPnPPortMapper = new UPnPPortMapper( ); } @Override protected Void dolnBackground(Void... params) { if(uPnPPortMapper != null) { try { Log. d(cg4u_log, Contacting Router for setting network configurations); if(uPnPPortMapper.openRouterPort(externallP, externalPort, internallP, internalPort, CG4UGames)) { Log.d(cg4u_log,String.format(Setting network configurations successful IP:%s Port:%d,externallP,externalPort)); Log.d(cg4u_log,String.format(Setting network configurations successful IP:%s Port:%d,intemallP, internalPort)); } } catch (lOException e) { e.printStackTrace( ); } catch (UPNPResponseException e) { e.printStackTrace( ); } } return null; } @Override protected void onPostExecute(Void result) { super.onPostExecute(result); / /Send broadcast for update in the main activity / /Intent i = new lntent(ApplicationConstants.APPLICATION_ENCODING_TEXT); / /context.sendBroadcast(i); } } .sub./****************************** UniversalPortMapper.java********************************* ********************************************************************************************* / package org.cloundgaming4u.client.portforwarding; import net.sbbi.upnp.impls.InternetGatewayDevice; import net.sbbi.upnp.messages. UPNPResponseException; import java.io. lOException; public class UPnPPortMapper { private lnternetGatewayDevice[ ] internetGatewayDevices; private InternetGateway Device foundGatewayDevice; I** * Search for IGD External Address * @return String */ public String findExternallPAddress ( ) throws lOException, UPNPResponseException { /** Upnp devices router search*/ if(internetGatewayDevices == null) { internetGatewayDevices = lnternetGatewayDevice.getDevices(ApplicationConstants.SCAN_TIMEOUT); } if(internetGatewayDevices != null) { for (InternetGatewayDevice IGD : internetGatewayDevices) { foundGatewayDevice = IGD; return IGD.getExternallPAddress( ).toString( ); } } return null; } I** * Search Found Internet Gateway Device Friendly Name * @ return */ public String findRouterName( ){ if(foundGatewayDevice != null){ return foundGatewayDevice. getlGDRootDevice( ).getFriendlyName( ).toString( ); } return null; } I** * Open Router Port * IGD == Internet Gateway Device * * @param internallP * @param internalPort * @param externalRouterIP * @param externalRouterPort * @param description * @ return * @throws lOException * @throws UPNPResponseException */ public boolean openRouterPort(String externalRouterIP, int externalRouterPort, String internallP, int internalPort, String description) throws lOException, UPNPResponseException { /** Upnp devices router search*/ if(internetGateway Devices == null){ internetGatewayDevices = lnternetGatewayDevice.getDevices(ApplicationConstants.SCAN_TIMEOUT); } if(internetGatewayDevices != null){ for (InternetGatewayDevice addIGD : internetGatewayDevices) { /** Open port for TCP protocol and also for UDP protocol * Both protocols must be open this is a MUST*/ //addlGD.addPortMapping(description, externalRouterIP, internalPort, externalRouterPort, internallP, 0, ApplicationConstants.TCP_PROTOCOL); addlGD.addPortMapping(description, externalRouterIP, internalPort, externalRouterPort, internallP, 0, ApplicationConstants.UDP_PROTOCOL); } return true; }else{ return false; } } public boolean removePort(String externallP.int port) throws lOException, UPNPResponseException{ /** Upnp devices router search*/ if(internetGatewayDevices == null){ internetGatewayDevices = lnternetGatewayDevice.getDevices(5000); } /**Remote port mapping for all routers*/ if(internetGatewayDevices != null){ for (InternetGatewayDevice removelGD : internetGatewayDevices) { / / removelGD. deletePortMapping(externallP, port, ApplicationConstants.TCP_PROTOCOL); removelGD.deletePortMapping(externallP, port, UDP); } return true; }else{ return false; } } } *********************************************** End of ClientNetworkCommunication
2. The method as claimed in claim 1, wherein the entire application is streamed on the local terminal without a time delay in parallel and almost in real time, while the relevant streamed application is downloaded, installed, continuously updated and launched locally on the local terminal and the stream is closed as soon as the files for the application have been made available in full on the local terminal, on which the application files are continuously updated.
3. The method as claimed in claim 1, wherein an additional telecommunication element is provided, which monitors the streaming process with a connection test device and the relevant streaming client and for the installation process a download device, an installation device and an update device are provided, wherein the connection test device works as follows: the connection test device is responsible for evaluating the connection between local client and streaming server, wherein the evaluation is carried out on the basis of four criteria: bandwidth, latency, hops, connection, wherein the terms mean: Bandwidth: in this case a file is downloaded from the streaming server, wherein the bandwidth is measured when downloading the files; Latency: in this case, the time taken by a signal to move from the streaming server to the client is measured; Hops: in this case, the connection node or nodes through which a signal must be passed before it reaches the client is/are measured; Connection: the relevant connection in this case is in relation to the packet loss, where the packet loss in optical transport media is lower than in copper cables; wherein the threshold values are individually configured and tailored to the requirements of the application, wherein in the event that the connection is not adequate, the streaming client will not be launched and the game or similar program is only loaded/installed/updated when the connection fulfils the predefined values; and that in the event that the connection test is successful, the streaming client is launched and the connection to the streaming server is set up and the latter streams the others directly to the local client, wherein the streaming client is active until the application is installed and fully executable on the terminal, wherein the sequence between the streaming server and the client takes place as follows: when setting up a direct connection, the following steps are initiated between the client and the relevant streaming server: i. recording the audio/video data of the running application, for example a game, via the streaming server on which the game is running. ii. compressing the audio/video data using high-quality hardware encoders; iii. transmission of the compressed audio/video data via WAN; iv. receiving the audio/video data by the telecommunication terminal; v. decompressing the audio/video data vi. reproduction of the audio/video data on the telecommunication terminal (small); vii. recording the actions (inputs) of the user of the telecommunication terminal, for example a player, on the telecommunication terminal (small); viii. efficient transmission of the inputs back to the relevant streaming server of the game and ix. playback of the transmitted inputs on the streaming server.
4. A telecommunications network for streaming and reproducing applications (APPs) via a particular telecommunications system, in which one or more streaming servers, which can be connected together by telecommunication, execute the relevant application and which form a connection to the respective telecommunication terminal which is characterized by an optimal connection, taking account of the number of hops, latency, bandwidth and ISP, wherein the relevant telecommunication terminal retrieves the desired application from a streaming server with optimal connection, which provides the processing power for the rendering and encoding of the relevant application, for the reproduction of applications on non-native application system environments, which differ ether due to different hardware or software components, wherein the streaming server undertakes the handling of the different applications and the rendering/encoding of the application and its audio and video signals, wherein the data are transmitted to the respective telecommunication terminalmobile wireless device, tablet, laptop, PC, TVand the transmission is performed using a modified h.264 protocol and the WAN is used as the transmission medium for audio/video packets via UDP/TCP and the entire processing power is used by the relevant streaming server, wherein the packet-based data are decoded only at the telecommunication terminal, wherein in order to provide a platform-independent streaming technology, which is programmed once and is portable to any telecommunication terminals, in which the streaming of individual applications, for example video games, is carried out over a WAN, in such a way that a) a communication with the session server is performed by means of the telecommunication terminal (small applications); b) a specific session is implemented for a specific end-user for the streaming server of the relevant application, for example a game, which is optimally nearest to the telecommunication terminal; c) session information is communicated to the telecommunication terminal and the streaming server by the relevant session server; d) a direct connection is implemented between the telecommunication terminal and the streaming server of the relevant application, for example a video game; e) when setting up a direct connection between the telecommunication terminal and the relevant streaming server the following steps are initiated: i. recording the audio/video data of the running application, for example a game, via the streaming server on which the game is running; ii. compressing the audio/video data using high-quality run-time modified hardware encoders; iii. transmission of the compressed audio/video data via WAN; iv. receiving the audio/video data by the telecommunication terminal; v. decompressing the audio/video data using a modified decoder; vi. reproduction of the audio/video data on the telecommunication terminal (small); vii. recording the actions (inputs) of the user of the telecommunication terminal, for example a player, on the telecommunication terminal (small) by a Thin Layer, which is placed dynamically over the display field; viii. efficient transmission of the inputs back to the relevant streaming server of the game and ix. reproduction of the transmitted inputs on streaming server, wherein via a particular telecommunications system, in which one or more streaming servers, which can be connected together by telecommunication, execute the relevant application and which form a connection to the respective telecommunication terminal which is characterized by an optimal connection, taking account of the number of hops, latency, bandwidth and ISP, wherein the relevant telecommunication terminal retrieves the desired application from a streaming server with optimal connection, which provides the processing power for rendering the video stream and the encoding of the relevant application, for the reproduction of applications on non-native application system environments, which differ either due to different hardware or software components, wherein the streaming server undertakes the handling of the different applications and the rendering/encoding of the application and its audio and video signals of the individual images (frames), wherein the data are transmitted to the respective telecommunication terminalmobile wireless device, tablet, laptop, PC, TVand the transmission is performed using a modified h.264 protocol and the WAN is used as the transmission medium for audio/video packets via UDP/TCP and the entire processing capacity is used by the relevant streaming server, wherein the packet-based data are decoded only at the telecommunication terminal, wherein the playback of applications is streamed on a local terminal independently of the downloading on the local terminal.
5. The telecommunications network as claimed in claim 4, wherein the telecommunications network is assigned at least one additional telecommunications element, which comprises a streaming process with a connection tester and at least one streaming client for the installation of the files and an element for downloading, installing and updating is provided for the application.
6. The telecommunications network as claimed in claim 4, wherein the entire application is streamed directly on the local terminal without a time delay, while the application actually being streamed is downloaded, installed, updated and launched locally on the local terminal and the stream is closed as soon as the files for the application have been made available in full on the local terminal.
7. The use of a telecommunications network for providing a platform-independent streaming technology, which is programmed once and is portable to any telecommunication terminals, in which the individual applications, for example video games, are streamed over a WAN, in such a way that a) a communication with the session server is performed by means of the telecommunication terminal (small applications); b) a specific session is implemented for a specific end-user for the streaming server of the relevant application, for example a game, which is optimally nearest to the telecommunication terminal; c) session information is communicated to the telecommunication terminal and the streaming server by the relevant session server; d) a direct connection is implemented between the telecommunication terminal and the streaming server of the relevant application, for example a video game; e) when setting up a direct connection between the telecommunication terminal and the relevant streaming server, the following steps are initiated: i. recording the audio/video data of the running application, for example a game, via the streaming server on which the game is running; ii. compressing the audio/video data using high-quality run-time modified hardware encoders; iii. transmission of the compressed audio/video data via WAN; iv. receiving the audio/video data by the telecommunication terminal; v. decompressing the audio/video data using a modified decoder; vi. reproduction of the audio/video data on the telecommunication terminal (small); vii. recording the actions (inputs) of the user of the telecommunication terminal, for example a player, on the telecommunication terminal (small) by a Thin Layer, which is placed dynamically over the display field; viii. efficient transmission of the inputs back to the relevant streaming server of the game and ix. playback of the transmitted inputs on the streaming server, wherein in the event of packet loss during the transmission of data to the telecommunication terminal, for example from a streaming server to the telecommunication terminal, making use of a plurality of specially developed error correction procedures, the following steps are carried out: a) recovery strategies are invoked in order to maintain a smooth gaming experience; b) the appropriate recovery strategy is selected, and c) the recovery request is sent back to the relevant streaming server of the application, for example the game, this process can also be initialized from the streaming server, wherein the telecommunications network is assigned at least one additional telecommunications link, preferably for each client, which comprises a streaming process with a connection tester and at least one streaming client for the installation of the files and an element for downloading, installing and updating is provided for the application.
8. (canceled)
Description
[0117]
[0118]
[0119]
[0120]
[0121]
[0122]
[0123]
[0124]
[0125]
[0126] The client, as a thin client, is used for the transmission of audio/video signals and can typically be deployed on any platform. A streaming server 120 can be in a relationship of 1:n, however, a client can only receive the communication to a particular streaming server 120. Typically, the number of clients per streaming server is not limited by the software but by the corresponding hardware capacities of the GPU of the streaming server 120.
[0127] Communication between the streaming server 120 and the client 110A is always initially established via the session management server 130. This performs the initial request of the client 110A to connect to the streaming server and searches for the optimal streaming server 120 for the client 110A. One system can have multiple streaming servers operating in parallel. They also do not always need to be in the same data center or country. After the assignment of a streaming server 120 by the session management server 130 for the client 110A, the streaming server 120 handles the direct communication with the client 110A.
[0128] A further element is the content server 195. This server is responsible for the delivery of certain parts in the interactive layer of the client 110A. Among other things, this controls the display of advertising according to the application that is displayed on the thin client. The necessary information is provided to the content server 195 via the session management server 30.
[0129] The communication takes place primarily via the Wide Area Network (WAN) 115. This includes various modes of transmission and is not limited to certain sections.
[0130]
[0131] The Port Authentication Module 230 handles the port authentication and corresponds to the provision of the connection between game stream server 120 and the client 110A. The Control Relay 240 is responsible for XXX. The task of the network relay 250 is to send off the corresponding packages and to manage incoming packets. The recovery module 260 is responsible for answering the corresponding recovery requests of the client 110A.
[0132]
[0133]
[0134] Client Session Manager 410, communicates with the streaming server 120 and the session management server and is initially responsible for the authentication and management of the client.
[0135] Network module 420 is responsible for configuring and maintaining the connection. This module also handles the sending and receiving of various packets.
[0136] The Controller 430 handles the dispatch of the delivered frames and audio packets as a visual image in the client.
[0137] Decode Render Video 440 and Decode Render Audio 450 receive the packets that were previously received from the network module 420 and were forwarded by the controller 430.
[0138] The elevator module 460 is responsible for collecting statistical data and transmits this to the session management server. Accordingly, the session management server can also optimize the connection. This results in a feedback, which makes this module very important.
[0139] The recovery module 470 evaluates incoming data packets. If a data packet turns out to have errors, the module selects a recovery strategy and, if necessary, requests a new packet from the streaming server or takes other measures to compensate for the loss, without suffering a loss in terms of latency or quality.
[0140] The client UI includes the interactive layer and content of the content server 195. This is where the input of the user is captured and sent off to the streaming server 120.
[0141]
[0142] The Content Administration is used for presetting e.g. the advertising to be displayed within the interactive layer in the client 110. The Content Administration 510 is designed to specify both the frequency as well as the content.
[0143] The Content Streaming module 520 handles the playback of the content and is used as a central interface for all clients.
[0144]
[0145] The application layer 610 reproduces the received frames and is responsible for the visual representation of the application.
[0146] On top of the application layer 610 is located the UI layer 620. This layer can be configured individually, but in principle, it is definitively responsible for the input by the user in the client.
[0147] In addition to the above-mentioned layers, there is the option to import content from the content server 195. This takes place in the area of the Content Layer 630.
[0148]
[0149] Once the decision is made, whether Blocking 730, Not Blocking 740, Intrarefresh 750 or Frame Validation 760 has been selected, the recovery request 770 is sent to the streaming server 120. The streaming server accordingly sends a new packet and the task of the recovery module 470 has been completed.
[0150]
[0151] During the streaming process, a connecting device 900 designed as a connection tester first tests the software to be loaded of the application for the relevant streaming client, for example 110A, while at the same time, in parallel with this, in our case a game is in progress via a gaming install-process. It can also, of course, be a different application. A download device designed as a downloader 1000 downloads the software onto the terminal device, while an installation device 2000 designed as an installer performs the installation at the same time, and an updating device designed as an updater 3000 always ensures that the latest data is available, thus performs a constant update process.
[0152] The device apparent from
[0153] The telecommunications element 800 thus allows two separate processes, namely the streaming process and the user install process, for example a gaming install process.
[0154] The streaming process in the telecommunications element 800 consists of two modules:
[0155] Connection tester: responsible for the testing of the connection on the basis of three criteria: Latency, bandwidth and network. In case the connection does not meet the requirements, the streaming client is not started. The threshold values for the three criteria can be flexibly adjusted and tailored to the application.
[0156] Streaming client: If the connection tester proves successful, then the streaming client will be started.
[0157] By using the streaming technology this streams the application directly into a local window of the terminal, for example into the PC window of a player.
[0158] During installation this process handles a plurality of tasks and processes them sequentially. This process is responsible for putting the application locally into an executable state, while it is being streamed. All steps are performed in the background of the system and do not require any input by the player. The streaming process performs the following steps:
[0159] Download: The application is downloaded onto the local computer in parallel with the stream.
[0160] Installation: The process installs the application in the background. Any required third-party software is also installed.
[0161] Updating: The application is updated.
[0162] Once these steps are completed, the user can run the application locally after restarting the telecommunications element 800.
[0163] The features of the invention disclosed in the description, the claims and those which are apparent from the drawings can be essential to the implementation of the invention both individually and in any desired combinations.
Explanation of Terms
[0164]
TABLE-US-00002 Application Layer Anwendungsebene Applications-Code Anwendungscode buffering Pufferung Client Ein Client (deutsch Kunde, auch clientseitige Anwendung oder Clientanwendung) bezeichnet A client (German Kunde, also client-side application or client application) refers to a computer program, which is executed on a terminal of a network and communicates with a central server. Client UI Module Client Benutzeroberflche Modul Client-Application Client-Anwendung Client-Session Client-Sitzung Cloud Complex of multiple servers in the Internet codec coder-decoder Content Layers Inhaltsebene Content Servers Inhalt-Server Content Streaming Inhalt Streaming Controller Regler Controller Module Regler-Modul decode thread decoding application Decoder-Render Dekodierungs-Ausfuhrer Evaluator evaluation unit Evaluator Modul Evaluierungsmodul Frame validation Frame berprfer interactive layer Interaktiv-Ebene Intrarefresh internal refresh/updating Loadbalancing Lastenverteilung Network Module Netzwerk-Modul Not Blocking Nicht-Blockieren Overlay berlagerung packaging Paketierung Recovery Module Wiederherstellungs-Modul Recovery Strategies Wiederherstellungs-Strategie Render Thread is responsible for the rendering [display] of the application. Scripting script language session handler Sitzungs-Manager shared library Geteilte Bibliotheken Streaming Streaming Streaming-Server Streaming-Server Streaming-Technology Streaming-Technologie Timestamping Describes the allocation of a piece of data to a data packet UDP User Datagram Protocol UI Layer Benutzeroberflche-Ebene WAN Wide Area Network Web-Apps Web-Applikationen Webcontent Management Web-Inhalt Management
List of Citations
[0165] WO 2009/073830 A1 [0166] WO 2010/141522 A1 [0167] WO 2012/037170 A1 [0168] US 2014/0073428 A1 [0169] WO 2009/073830 A1 [0170] WO 2010/141522 A1 [0171] US 2015/018094 A1 [0172] WO 2014/186858 A1 [0173] Anonymous: CloudgamingWikipedia, the free encyclopedia, 24. Jun. 2015 (2015-06-24), XP055268633, found on internet: [0174] URL:https://en.wikipedia.orq/w/index.php?title=Cloudqam inq&oldid=668477716 [0175] [accessed on 2016-04-26] Section Types, 2nd paragraph; page 1