CLOUD PLATFORM CAPABLE OF PROVIDING REAL-TIME STREAMING SERVICES FOR HETEROGENEOUS APPLICATIONS INCLUDING AR, VR, XR, AND MR IRRESPECTIVE OF SPECIFICATIONS OF HARDWARE OF USER

Abstract

Proposed is a cloud platform for realizing all 5G services including high-quality heterogeneous applications including augmented reality (AR), virtual reality (VR), extended reality (XR), and mixed reality (MR) or mobile content based on the cloud and providing real-time streaming of these services irrespective of specifications of hardware of users.

Claims

1. A cloud platform capable of providing a real-time streaming service for heterogeneous applications including augmented reality (AR), virtual reality (VR), extended reality (XR), and mixed reality (MR) irrespective of specifications of hardware of a user, the cloud platform comprising: an image input unit 10 configured to receive an image captured by a terminal of a user which is able to capture images, to process the received image through non-compression conversion, and to transmit the processed image to an application program unit; a controller 20 configured to receive control event data input through a control terminal of the user, to analyze the control event data, and to transmit a driving control signal to the application program unit; wherein the application program unit 30 is configured to select and execute one of the heterogeneous applications according to the control signal transmitted from the controller 20 and to combine a virtual image realized by the selected application and the processed image transmitted from the image input unit 10 to generate an image content; a result output unit 40 configured to receive the image content generated through the application unit 30, to convert the image content through compression, and to stream the image content to an image terminal of the user which is able to reproduce images in real time through a streaming engine; and a data synchronization processor 50 configured to synchronize the image input unit 10, the controller 20, and the result output unit 40 with the application program unit 30.

2. The cloud platform according to claim 1, wherein the image input unit 10 includes: an AV receiver 101 configured to receive the image captured by the terminal of the user; an AV non-compression converter 102 configured to convert the image received through the AV receiver 101 according to the non-compression conversion; and a transmitter 103 configured to transmit the image converted through the AV non-compression converter 102 to the application program unit through the data synchronization processor.

3. The cloud platform according to claim 1, wherein the controller 20 includes: a control event receiver 201 configured to receive a control event input through the control terminal of the user; a data analyzer 202 configured to analyze data of the control event received through the control event receiver 201; and a transmitter 203 configured to transmit the data analyzed through the data analyzer 202 to the application program unit through the data synchronization processor.

4. The cloud platform according to claim 1, wherein the application program unit 30 includes a plurality of image content generators 31, wherein each image content generator 31 includes: an AV input processor 311 configured to receive the image transmitted from the image input unit 10; a control input processor 312 configured to receive the control signal transmitted from the controller 20; the application 313 selected from the plurality of heterogeneous; and a result processing and generation unit 314 configured to combine the processed image input through the AV input processor 311 and the virtual image generated by the application 313 according to the control signal of the control input processor 312 to generate the image content, to synchronize the image content, and to transmit the synchronized image content to the result output unit 40.

5. The cloud platform according to claim 1, wherein the result output unit 40 includes: an output receiver 401 configured to receive the image content generated through the application program unit 30; an AV compression converter 402 configured to compress the image content received through the output receiver 410; and a streaming engine 404 configured to stream the image transmitted through the AV compression converter 402 to the terminal of the user.

6. The cloud platform according to claim 5, wherein the result output unit 40 further includes a transmission packet converter 403.

7. The cloud platform according to claim 2, wherein the AV non-compression converter 102 processes the received image according to graphics processing unit (GPU) based parallel processing.

8. The cloud platform according to claim 5, wherein the AV compression converter 402 processes the image content according to graphics processing unit (GPU) based parallel processing.

9. A cloud platform capable of providing a real-time streaming service for heterogeneous applications including augmented reality (AR), virtual reality (VR), extended reality (XR), and mixed reality (MR) irrespective of specifications of hardware of a user, the cloud platform comprising: an image input unit 10 configured to receive an image captured by a user, to process the received image through non-compression conversion, and to transmit the processed image to an application program unit; a controller 20 configured to receive control event data input through a control terminal of the user, to analyze the control event data, and to transmit a driving control signal to the application program unit; wherein the application program unit 30 includes a plurality of image content generators 31 configured to select and execute one of the heterogeneous applications according to the control signal transmitted from the controller 20 and to combine a virtual image realized by the selected application and the processed image transmitted from the image input unit 10 to generate an image content, and wherein each image content generator 31 includes: an AV input processor 311 configured to receive the image transmitted from the image input unit 10; a control input processor 312 configured to receive the control signal transmitted from the controller 20; the application 313 selected from the plurality of heterogeneous applications ; and a result processing and generation unit 314 configured to combine the processed image input through the AV input processor 311 and the virtual image generated by the application 313 according to the control signal of the control input processor 312 to generate the image content, to synchronize the image content, and to transmit the synchronized image content to the result output unit 40; a result output unit 40 configured to receive the image content generated through the application unit 30, to convert the image content through compression, and to stream the image content to an image terminal of the user in real time through a streaming engine, and a data synchronization processor 50 configured to synchronize the image input unit 10, the controller 20, and the result output unit 40 with the application program unit 30.

10. A cloud platform capable of providing a real-time streaming service for heterogeneous applications including augmented reality (AR), virtual reality (VR), extended reality (XR), and mixed reality (MR) irrespective of specifications of hardware of a user, the cloud platform comprising: an image input unit 10 configured to receive an image captured by a user, to process the received image through non-compression conversion, and to transmit the processed image to an application program unit; a controller 20 configured to receive control event data input through a control terminal of the user, to analyze the control event data, and to transmit a driving control signal to the application program unit; wherein the application program unit 30 is configured to select and execute one of the heterogeneous applications according to the control signal transmitted from the controller 20 and to combine a virtual image realized by the application and the processed image transmitted from the image input unit 10 to generate an image content; a result output unit 40 configured to receive the image content generated through the application unit 30, to convert the image content through compression, and to stream the image content to an image terminal of the user in real time through a streaming engine and including an output receiver 401 configured to receive the image content generated through the application program unit 30, an AV compression converter 402 configured to compress the image content received through the output receiver 410, and a streaming engine 404 configured to stream the image content transmitted through the AV compression converter 402 to a terminal of the user in real time; and a data synchronization processor 50 configured to synchronize the image input unit 10, the controller 20, the result output unit 40 with the application program unit 30.

Description

DESCRIPTION OF DRAWINGS

[0015] FIG. 1 is a configuration diagram of a cloud platform according to the present invention.

[0016] FIG. 2 is a configuration of an image input unit constituting the cloud platform according to the present invention.

[0017] FIG. 3 is a configuration of a controller constituting the cloud platform according to the present invention.

[0018] FIG. 4 is a configuration of an application program unit constituting the cloud platform according to the present invention. FIG. 5 is a configuration of a result output unit constituting the cloud platform according to the present invention.

[0019] FIG. 6 is a configuration diagram illustrating an operation flow of the cloud platform according to the present invention.

BEST MODE

[0020] A cloud platform capable of providing a real-time streaming service for heterogeneous applications including AR, VR, XR, and MR irrespective of specifications of hardware of a user includes an image input unit 10 configured to receive an image captured by a terminal of a user which is able to capture images, to process the received image through non-compression conversion, and to transmit the processed image to an application program unit, a controller 20 configured to receive control event data input through a control terminal of the user, to analyze the control event data, and to transmit a driving control signal to the application program unit, the application program unit 30 being configured to select and execute one of the heterogeneous applications including AR, VR, XR, and MR according to the control signal transmitted from the controller 20 and to combine a virtual image realized by the application and the actual processed image transmitted from the image input unit 10 to generate an image content, a result output unit 40 configured to receive the image content generated through the application unit 30, to convert the image content through compression, and to stream the image content to a user's image terminal capable of reproducing images in real time through a streaming engine, and a data synchronization processor 50 configured to synchronize the image input unit 10, the controller 20, and the result output unit 40 with the application program unit 30.

Mode for Invention

[0021] A cloud platform capable of providing a real-time streaming service for heterogeneous applications including AR, VR, XR, and MR irrespective of specifications of hardware of a user according to the present invention will be described with reference to the attached drawings.

[0022] As illustrated in FIG. 1 and FIG. 6, a cloud platform 1 according to the present invention includes an image input unit 10 configured to receive an image captured by a terminal of a user which is able to capture images, to process the image through non-compression conversion, and to transmit the processed image to an application program unit, a controller 20 configured to receive control event data input through a control terminal of the user, to analyze the control event data, and to transmit a driving control signal to the application program unit, the application program unit 30 being configured to select and execute one of the heterogeneous applications including AR, VR, XR, and MR according to the control signal transmitted from the controller 20 and to combine a virtual image realized by the application and the actual processed image transmitted from the image input unit 10 to generate an image content, a result output unit 40 configured to receive the image content generated through the application unit 30, to convert the image content through compression, and to stream the image content to a user's image terminal capable of reproducing images in real time through a streaming engine, and a data synchronization processor 50 configured to synchronize the image input unit 10, the controller 20, and the result output unit 40 with the application program unit 30.

[0023] The cloud platform 1 of the present invention including the aforementioned configuration may combine graphics-intensive AR, VR, XR, and MR application content that requires specific hardware and OS environment, such as games, and a real-time image captured by a user to generate image content irrespective of specifications of hardware of the user and stream a resultant image through a user terminal, such as an IPTV, a mobile terminal, a set-top box (STB), or a PC, in real time.

[0024] The user may easily control applications using external input devices such as a joystick, a keyboard, a mouse, a remote controller, or a gesture.

[0025] The cloud platform 1 having such a function is applicable to various fields such as a graphics-intensive game content field, an education content field, a print media content field, a web content field, and a mobile content field.

[0026] For example, in the case of the game content field, content directly created by a user may be converted into realistic 3D content and inserted into a game image based on the cloud, or a user action may be recognized and a game may be played according to the user action in a game scene based on the cloud according to the cloud platform 1 of the present invention.

[0027] In the case of the print media content field, print media such as magazines and books and AR may be combined to overcome flat and unilateral characteristics and to extend the print media concept.

[0028] In the case of an advertisement content field, advertisement related characters and graphics may be combined with the real world and displayed to provide more realistic advertisements to users.

[0029] Hereinafter, components constituting the cloud platform 1 will be described.

[0030] 1. Image Input Unit 10

[0031] The image input unit 10 receives an image captured through a terminal of a user, processes the image through non-compression conversion and transmits the processed image to the application program unit.

[0032] As illustrated in FIG. 1, FIG. 2, and FIG. 6, the image input unit 10 includes an AV receiver 101 that receives an image captured by a terminal of a user which can capture images in real time, an AV non-compression converter 102 that converts the image received through the AV receiver 101 according to non-compression conversion, and a transmitter 103 that transmits the image converted through the AV non-compression converter 102 to the application program unit through the data synchronization processor.

[0033] Examples of the user terminal used to transmit an image to the AV receiver 101 include a smartphone camera, a web camera, a VDSLR, a video camera, and 360VR equipment. The AV receiver 101 receives an image transmitted from the user terminal in real time, decodes the received image into video and audio data in real time through the AV non-compression converter 102, and transmits the video and audio data to the application program unit through the transmitter 103.

[0034] The AV non-compression converter 102 may perform rapid processing through GPU based parallel processing, and according to this configuration, an image captured by the user and input in real time may be combined with a virtual image of an application and displayed on a monitor of the user in real time.

[0035] In addition, since operation from combination of the image input by the user in real time and a virtual image of the application to final output of the combined image is performed within tens of milliseconds (ms) according to configurations of the AV non-compression converter 102 of the image input unit 10 and an AC compression converter 403 of the result output unit 40, real-time streaming of the image can be performed such that the user scarcely feels a time difference between the input image and the output image.

[0036] According to the cloud platform 1 including the aforementioned configuration, large-capacity images can also be processed in real time and thus the user can easily experience real-time streaming image content provided through the cloud platform 1 in a state in which there is no burden on a response speed.

[0037] 2. Controller 20

[0038] The controller 20 receives control event data input through a control terminal of the user, analyzes the control event data, and transmits an application driving control signal. As illustrated in FIG. 1, FIG. 3, and FIG. 6, the controller 20 includes a control event receiver 201 that receives a control event input through the control terminal of the user, a data analyzer 202 that analyzes data of the control event received through the control event receiver 201, and a transmitter 203 that transmits data analyzed through the data analyzer 202 to the application program unit through the data synchronization processor.

[0039] Specific examples of the control terminal of the user which transmits a control event to the event receiver 201 include external input devices such as a joystick, a keyboard, a mouse, and a remote controller.

[0040] 3. Application Program Unit 30.

[0041] The application program unit 30 selects and executes one of the heterogeneous applications including AR, VR, XR, and MR according to the control signal transmitted from the controller 20 and combines a virtual image of the executed application and the actual image transmitted from the image input unit 10 to generate image content.

[0042] As illustrated in FIG. 1, FIG. 4, and FIG. 6, the application program unit 30 includes a plurality of image content generators 31, and each image content generator 31 includes an AV input processor 311 that receives an image transmitted from the image input unit 10, a control input processor 312 that receives a control signal transmitted from the controller 20, an application 313 selected from the AR, VR, XR, and MR heterogeneous applications, and a result processing and generation unit 314 that combines the actual image input through the AV input processor 311 and a virtual image generated by the application 313 according to the control signal of the control input processor 312 to generate an image content, synchronizes the image content and transmits the synchronized image content to the result output unit 40. The present invention can provide a plurality of heterogeneous applications including AR, VR, XR, and MR in an integrated manner through a single platform based on the cloud according to a control signal of a user, and thus the user can receive a real-time streaming service of a plurality of heterogeneous applications including AR, VR, XR, and MR.

[0043] That is, the cloud platform 1 according to the present invention can provide choices with respect to high-quality AR, VR, XR, and MR applications or mobile content in the form of a menu through a terminal of a user, and the user can select a desired program and then immediately execute the program to be provided with an image content service in real time through his/her terminal without downloading and installing a large-capacity file.

[0044] 4. Result Output Unit 40

[0045] The result output unit 40 receives the image content generated through the application program unit 30, converts the image content through compression, and then streams the compressed image content to the user terminal in real time through a streaming engine.

[0046] As illustrated in FIG. 1, FIG. 5, and FIG. 6, the result output unit 40 includes an output receiver 401 that receives the image content generated through the application program unit 30, an AV compression converter 402 that compresses the image content received through the output receiver 410, and a streaming engine 404 that stably streams the image transmitted through the AV compression converter 402 to the user terminal.

[0047] Here, the result output unit 40 may further include a transmission packet converter 403.

[0048] The AV compression converter 402 may rapidly process an image through GPU based parallel processing, and image content rapidly processed in this manner is provided to the user terminal through real-time streaming using the streaming engine 404.

[0049] Since the image content provided through the streaming engine 404 is implemented based on the cloud and streamed in real time, the user can stream and experience graphics-intensive content in real time using an existing IPTV, mobile terminal, set-top box (STB), or a PC even when he/she does not have a terminal with advanced specifications.

[0050] 5. Data Synchronization Processor 50

[0051] The data synchronization processor 50 is provided between the image input unit 10, the controller 20, the result output unit 40, and the application program unit 30, as illustrated in FIG. 1 and FIG. 6, and synchronizes data signals transmitted/received therebetween.

INDUSTRIAL APPLICABILITY

[0052] The cloud platform capable of providing a real-time streaming service for heterogeneous applications including AR, VR, XR, and MR according to the present invention based on the cloud can allow a user to easily execute and operate high-quality AR, VR, XR, and MR applications or mobile content through an IPTV, a mobile terminal, a set-top box, or a PC of the user without having a burden on data capacity and restraints on use according to heating and can allow service providers (Korea's mobile carriers such as KT, SKT, and LG Uplus, broadcasting companies, service providers that plan new services using new content, etc.) to construct a platform according to the present invention such that application programs developed by application developers can be directly used by users to remarkably reduce efforts and costs required to individually launch applications, and thus the present invention is industrially applicable.