SYSTEM, APPARATUS AND METHOD FOR PROVIDING ADAPTIVE AR STREAMING SERVICE

Abstract

A system for providing an adaptive AR streaming service may comprise: a cloud AR server that stores an AR object composed of a plurality of progressive mesh chunks, and a meta file including property information of the AR object and the progressive mesh chunks; and a mobile augmented reality (AR) device that requests and receives the meta file from the cloud AR server, parses the meta file to identify a property of the AR object, and implements augmented reality.

Claims

1. A system for providing an adaptive AR streaming service, comprising: a cloud AR server that stores an AR object composed of a plurality of progressive mesh chunks, and a meta file including property information of the AR object and the progressive mesh chunks; and a mobile augmented reality (AR) device that requests and receives the meta file from the cloud AR server, parses the meta file to identify a property of the AR object, and implements augmented reality.

2. The system for providing an adaptive AR streaming service according to claim 1, wherein the system uses a method for providing an AR streaming service with low latency and high quality in a time-varying unstable network by applying progressive mesh-based AR content to improve scalability and effectively compressing and providing quality information according to a distance, rotation angle, and LOD of a 3D object displayed on a screen of a real AR content.

3. The system for providing an adaptive AR streaming service according to claim 1, wherein the system uses a chunk scheduling method to improve visual perceptual quality of the AR streaming service by using a method for effectively compressing and providing quality information.

4. The system for providing an adaptive AR streaming service according to claim 1, wherein the system uses a root mean square error (RMSE) that is a value weighted by displaying a scale value of the AR object on a screen as an objective measure of spatial quality.

5. The system for providing an adaptive AR streaming service according to claim 1, wherein in the system, an initial latency is defined as a time interval required to display all AR objects on a screen regardless of spatial quality, and is adopted as a measure of temporal quality.

6. The system for providing an adaptive AR streaming service according to claim 1, wherein the cloud AR server is designed with a meta file structure in which a basic structure of the meta file is HTTP HAS (Adaptive Streaming).

7. The system for providing an adaptive AR streaming service according to claim 1, wherein the mobile augmented reality (AR) device includes a camera that continuously detects an AR marker to generate a camera image.

8. The system for providing an adaptive AR streaming service according to claim 1, wherein the mobile augmented reality (AR) device includes a marker detector that identifies a marker ID, coordinate, and a corresponding URL of the AR object when an AR marker is detected by continuously sensing the AR marker in a camera image.

9. The system for providing an adaptive AR streaming service according to claim 1, wherein the mobile augmented reality (AR) devices include a network monitor that continuously collects network status and estimates available throughput in the future.

10. The system for providing an adaptive AR streaming service according to claim 1, wherein the mobile augmented reality (AR) device includes a chunk scheduler that selects and requests a next chunk in order to maximize human visual perceptual quality.

11. The system for providing an adaptive AR streaming service according to claim 1, wherein the mobile augmented reality (AR) device includes an object renderer that renders a matching AR object and displays it on a screen when a requested chunk arrives.

12. An apparatus for providing an adaptive AR streaming service comprising: a processor; and a memory that stores at least one instruction executed by the processor, wherein the at least one instruction is configured to cause the processor to perform the steps of: reducing a spatial quality information overhead to provide an AR streaming service with low latency and high quality in a time-varying unstable network by applying progressive mesh-based AR content to improve scalability and effectively compressing and providing quality information according to a distance, rotation angle, and LOD of a 3D object displayed on a screen of a real AR content; and a chunk scheduling process to improve visual perceptual quality of the AR streaming service by utilizing a method for effectively compressing and providing quality information.

13. A method for providing an adaptive AR streaming service comprising the steps of: reducing a spatial quality information overhead to provide an AR streaming service with low latency and high quality in a time-varying unstable network by applying progressive mesh-based AR content to improve scalability and effectively compressing and providing quality information according to a distance, rotation angle, and LOD of a 3D object displayed on a screen of a real AR content; and a chunk scheduling process to improve visual perceptual quality of the AR streaming service by utilizing a method for effectively compressing and providing quality information.

14. The method for providing an adaptive AR streaming service according to claim 13, wherein the step of reducing a spatial quality information overhead comprises the steps of: (a) sampling with a discrete value according to Equation 5 since the rotation angle and the distance are continuous real numbers; (b) downsampling and then decomposing an RMS matrix with SVD in order to reduce a dimension of the RMS matrix according to Equations 6, 7, 8, and 9; and (c) reconstructing the RMS matrix through inverse SVD and upscaling using only large singular value and related vector according to Equation 10 and using the reconstructed RMS matrix for the chunk scheduling, $\begin{array}{cc}{\mathrm{RMS}}_{i,l,k}=\left[\begin{array}{ccc}{\mathrm{rms}}_{1,1}^{i,l,k}& .Math.& {\mathrm{rms}}_{1,.Math.P.Math.}^{i,l,k}\\ .Math.& \ddots & .Math.\\ {\mathrm{rms}}_{.Math.Y.Math.,1}^{i,l,k}& .Math.& {\mathrm{rms}}_{.Math.Y.Math.,.Math.P.Math.}^{i,l,k}\end{array}\right]& \left[\mathrm{Equation}5\right]\end{array}$ $\begin{array}{cc}\mathrm{Down}\left(n^{\mathrm{scale}},{\mathrm{RMS}}_{i,l,k}\right)=U_{i,l,k}{S}_{i,l,k}{V}_{i,l,k}^T& \left[\mathrm{Equation}6\right]\end{array}$ $\begin{array}{cc}{U}_{i,l,k}=\left[u_1^{i,l,k}.Math.u_{\frac{.Math.P.Math.}{n^{\mathrm{scale}}}}^{i,l,k}\right]& \left[\mathrm{Equation}7\right]\end{array}$ $\begin{array}{cc}{S}_{i,l,k}=\mathrm{diag}\left({\delta }_1^{i,l,k},{\delta }_2^{i,l,k},.Math.,{\delta }_{n_{\max }^{\mathrm{eigen}}}^{i,l,k}\left(\right)\right)& \left[\mathrm{Equation}8\right]\end{array}$ $\begin{array}{cc}{V}_{l,k}=\left[v_1^{l,k}.Math.v_{\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}}^{l,k}\right]& \left[\mathrm{Equation}9\right]\end{array}$ $\begin{array}{cc}{\mathrm{RMS}}_{i,l,k}^{\mathrm{rec}}=\mathrm{UP}\left(n^{\mathrm{scale}},\underset{m=1}{\stackrel{n^{\mathrm{eign}}}{.Math.}}{u}_m^{i,l,k}{\delta }_m^{i,l,k}{v}_m^{i,l,k^T}\right)& \left[\mathrm{Equation}10\right]\end{array}$ wherein, at the distance k, when rotated n.sub.th (∈P) about the z-axis and rotated by the min angle (∈Y) around the x-axis, rms.sub.m,n.sup.i,l,k is the RMS between an image displayed on a screen of an AR terminal when AR content #i renders a chunk of LOD #, and an image displayed on the screen of the AR terminal when the AR content #i renders a maximum LOD chunk, Down is an operator that reduces the RMS matrix in the |P|×|Q| dimension to $\frac{.Math.P.Math.}{n^{\mathrm{scale}}}⨯\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}$ by using downsampling, U.sub.i,l,k and V.sub.i,l,k are rectangular orthogonal matrixes that satisfy U.sub.i,l,kU.sub.i,l,k.sup.T=I and V.sub.i,l,kV.sub.i,l,k.sup.T=I, S.sub.i,l,k is a diagonal matrix composed of a square of a singular value in which a diagonal element is arranged in a descending order, n.sub.max.sup.eigen has a value of min $\left(\frac{.Math.P.Math.}{n^{\mathrm{scale}}},\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}\right),$ u.sub.m.sup.i,l,k and v.sub.m.sup.i,l,k are orthogonal eigenvectors associated with the singular value, UP is an upscaling operator that increases the size of the matrix by n.sup.scale times, and n.sup.eigen is the number of singular values used to reconstruct the RMS matrix through the upscaling.

15. The method for providing an adaptive AR streaming service according to claim 14, wherein in the step (a), among the sampled rotation angles, a set of key rotation angles rotating around the x-axis is defined as P, and a set of rotation angles rotating around the z-axis is defined as Y, RMS information at a distance for LOD #1 is expressed as in Equation 5, the RMS matrix uses downsampling and singular value decomposition (SVD) to reduce the size of the RMS matrix.

16. The method for providing an adaptive AR streaming service according to claim 14, wherein in the step (b), the RMS matrix reduced by the downsampling is decomposed into S, U, and V matrices through the SVD process, in the matrix decomposed through the SVD, more information of an original RMS matrix is included in the singular value with high value and related vector.

17. The method for providing an adaptive AR streaming service according to claim 14, wherein in the step (c), n.sup.eigen and n.sup.scale, which guarantee a R-squared value between an original RMS matrix, RMS.sub.i,l,k and the RMS.sub.i,l,k.sup.rec generated through the S, U, and V matrices generated in the step (b) and the upscaling, is higher than 0.95, are determined through global search, RMS information according to the rotation angle included in P and Y is transmitted by providing only the n.sup.eigen singular values and related eigenvectors, not the entire RMS matrix in a mobile terminal, when the 3D object is rendered and playbacked by rotating at an angle not belonging to P and Y, an RMS value at an adjacent rotation angle included in P and Y is obtained by linear interpolation.

18. The method for providing an adaptive AR streaming service according to claim 13, wherein the chunk scheduling process comprises the steps of: (d) initializing {right arrow over (l)}.sup.trans to 0; (e) transmitting c.sub.i,j.sup.byte (for 1≤i≤N.sub.AR and 1≤j≤L.sub.i.sup.max ( )) and S, V, and D values from an AR server, and reconstructing a RMS matrix through the S, V, and D; (f) at an AR terminal, identifying {right arrow over (θ)}, {right arrow over (ψ)}, {right arrow over (d)}, and {tilde over (t)}r.sub.tcp; (g) checking whether or not an expected download time for LOD #l.sub.i.sup.trans+1 chunk of AR content #i exceeds T.sup.max, and going to step (i) if one or more chunks excess; (h) after selecting the AR content #i that maximizes Equation 12, going to step (j); (i) when receiving the LOD #l.sub.i.sup.trans+1 chunk, selecting the AR content #i that minimizes Equation 15 considering a tradeoff between WRMS and transmission delay; (j) requesting a next chunk of the selected AR content #i; (k) when the requested chunk is received, updating l.sub.i.sup.trans of {right arrow over (l)}.sup.trans with l.sub.i.sup.trans+1; and (l) repeatedly performing the steps (f) to (k) until all progressive mesh chunks are received, wherein (Equation 15) $\left(1-\alpha \right)\left(1-\frac{\omega \left(i,\stackrel{.fwdarw.}{d}\right)\left({\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}\right)-{\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}+1\right)\right)}{{.Math.}_{i=1}^{N_{\mathrm{AR}}}\omega \left(i,\stackrel{.fwdarw.}{d}\right)\left({\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}\right)\right)}\right)+\alpha .Math.\max \left(\frac{\left(T^{\mathrm{con}}+\frac{c_{i,l_i^{\mathrm{trans}}+1}^{\mathrm{byte}}}{{\mathrm{tcp}}_{}}\right)-T^{\max }}{T^{\max },0}\right).$

19. A computer program stored in a computer-readable recording medium for implementing the method for providing an adaptive AR streaming service according to claim 13.

20. A computer-readable recording medium for implementing a program of the method for providing an adaptive AR streaming service according to claim 13.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0034] FIGS. 1 and 2 are schematic diagrams of a system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0035] FIG. 3 is a conceptual diagram for explaining the rotation of an object according to the position of the camera 250 of the system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0036] FIG. 4 is a graph for explaining RMS according to the rotation angle, distance, and LOD of a Gargo 3D object in a method for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0037] FIG. 5 is an XML structure diagram of a proposed meta file of a system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0038] FIG. 6 is a configuration diagram of an apparatus 1000 for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Exemplary embodiments of the present disclosure are disclosed herein. However, specific structural and functional details disclosed herein are merely representative for purposes of describing exemplary embodiments of the present disclosure. Thus, exemplary embodiments of the present disclosure may be embodied in many alternate forms and should not be construed as limited to exemplary embodiments of the present disclosure set forth herein.

[0040] Accordingly, while the present disclosure is capable of various modifications and alternative forms, specific exemplary embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the present disclosure to the particular forms disclosed, but on the contrary, the present disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure. Like numbers refer to like elements throughout the description of the figures.

[0041] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

[0042] It will be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (i.e., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.).

[0043] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this present disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0045] Hereinafter, exemplary embodiments of the present disclosure will be described in greater detail with reference to the accompanying drawings. In order to facilitate general understanding in describing the present disclosure, the same components in the drawings are denoted with the same reference signs, and repeated description thereof will be omitted.

[0046] FIGS. 1 and 2 are schematic diagrams of a system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0047] Referring to FIGS. 1 and 2, an object of the system for providing an adaptive AR streaming service according to an embodiment of the present disclosure is to provide a high-quality adaptive AR streaming service through fast interaction over a time-varying wireless network.

[0048] A system for providing an adaptive AR streaming service according to an embodiment of the present disclosure includes a cloud AR server 100, a network emulator 130, an Internet 150, a wireless access point (AP), and a mobile augmented reality (AR) device 200, and uses a progressive mesh to improve the scalability of the AR streaming service.

[0049] The cloud AR server 100 stores an AR object 110 and a corresponding meta file 120. The AR object is composed of several progressive mesh chunks, and the meta file includes the AR object and the property information of the progressive mesh chunk 130. The meta file plays an important role in distributing the properties of the progressive mesh chunk 130 to the mobile AR device. The basic structure is designed by referring to the meta file structure of HTTP HAS (Adaptive Streaming).

[0050] The mobile augmented reality (AR) device 200 includes a marker detector 210, a network monitor 220, a chunk scheduler 230, an object renderer 240, and a wireless network interface 250.

[0051] The marker detector 210 continuously detects an AR marker 10 in the image of the camera 250. When the AR marker 10 is detected, the marker detector 210 identifies a marker ID, coordinates, and a corresponding URL of the AR object. Then, the mobile augmented reality (AR) device 200 requests and receives the meta file 120 from the cloud AR server 100. The meta file 120 is parsed to identify the AR object properties.

[0052] At the same time, the network monitor 220 continuously collects network status and estimates usable throughput in the future.

[0053] Using such information, the chunk scheduler 230 selects and requests the next chunk in order to maximize human visual perceptual quality. When the requested chunk arrives, the object renderer 240 renders the matching AR object and displays it on the screen. In fact, the human visual perceptual quality of AR streaming service depends on both spatial quality and temporal quality at the same time.

[0054] In the system for providing an adaptive AR streaming service according to an embodiment of the present disclosure, a root mean square error (RMSE), a value weighted by displaying a scale value of an AR object on a screen, is used as an objective measure of spatial quality. In addition, initial latency is defined as the time interval required to display all AR objects on the screen, regardless of their spatial quality, and is adopted as a measure of temporal quality.

[0055] FIG. 3 is a conceptual diagram for explaining the rotation of an object according to the position of the camera 250 of the system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0056] Referring to FIG. 3, in the present disclosure, the marker 10 may be used to identify the location where the AR content should be rendered and playbacked and its shape.

[0057] The present disclosure assumes that all the markers 10 are located on a plane including a floor and walls.

[0058] When the marker 10 is identified by the camera 250 of the AR terminal 200, the plane position of the AR content can be expressed as follows.

[00001]$\begin{array}{cc}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& \gamma & c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{1,1}& r_{1,2}& r_{1,3}& t_{1,1}\\ r_{2,1}& r_{2,2}& r_{2,3}& t_{2,1}\\ r_{3,1}& r_{3,2}& r_{3,3}& t_{3,1}\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]& \left[\mathrm{Equation}1\right]\end{array}$

[0059] In Equation 1 above, x, y and z are the coordinates of a 3D point in a world coordinate space, u and v are the pixel coordinates of the image plane projected on the screen of a user terminal, and f.sub.x and f.sub.y are focal lengths, c.sub.x and c.sub.y are principal points. γ is a skew parameter. f.sub.x, f.sub.y, c.sub.x, c.sub.y and γ can be obtained by camera calibration. The rotation and translation elements (r.sub.i,j and t.sub.i,j) of the object in Equation 1 can be obtained using AR and vision processing libraries such as ARToolkit, ARKit, ARCore, and OpenCV. Then, the detected marker rotation angle and distance can be calculated as follows.

[00002]$\begin{array}{cc}\theta ={\sin }^{-1}\left(-r_{2,3}\right)& \left[\mathrm{Equation}2\right]\end{array}$$\begin{array}{cc}\psi ={\tan }^{-1}\left(\frac{r_{2,1}}{r_{2,2}}\right)& \left[\mathrm{Equation}3\right]\end{array}$$\begin{array}{cc}d=\sqrt{t_{1,1}^2+t_{2,1}^2+t_{3,1}^2}& \left[\mathrm{Equation}4\right]\end{array}$

[0060] In the above equation, θ and ψ denote a rotation angle about the x-axis and z-axis, respectively, d is the distance between the mobile AR device 200 and the center of the marker 10. The size of the marker 10 on the screen is inversely proportional to d. The human visual perceptual quality of rendered AR content is related to the rotation angles (θ and ψ), distance and LOD.

[0061] FIG. 4 is a graph for explaining RMS according to the rotation angle, distance, and LOD of a Gargo 3D object in a method for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0062] Referring to FIG. 4, in the RMS according to the rotation angle, distance, and LOD of the Gargo 3D object in the method for providing an adaptive AR streaming service according to an embodiment of the present disclosure, (a) Distance is 45 cm, LOD #1, (b) distance is 45 cm, LOD #2, (c) distance is 60 cm, LOD #1 and (d) distance is 60 cm, LOD #2.

[0063] In the present disclosure, the 3D object uses the root mean square (RMS) of the image projected on the screen of the mobile AR device as an objective quality. When a mobile AR device requests multiple AR contents over a wireless network with limited bandwidth, the LOD chunk of the next requested AR content should be carefully selected to improve the user's visual quality according to the object's rotation angle and the distance of the AR content. The RMS of AR content is required for chunk scheduling to provide a low latency and high quality service, but since the RMS according to all rotation angles, distances, and LODs are very different as shown in FIG. 2, providing the RMS information for all situations is a great burden on the wireless network with limited bandwidth.

[0064] In order to solve this problem, the present disclosure proposes a compression technique of spatial quality information capable of efficiently transmitting RMS information with low overhead and a scheduling method using the same.

[0065] 1) A Method for Reducing Spatial Quality Information Overhead

[0066] Depending on the rotation of the 3D object, the distance from the camera, and the change of the LOD, the quality displayed in the actual AR terminal is very different. The present disclosure proposes a process of reducing spatial quality information overhead according to the rotation of the 3D object, the distance, and the LOD in order to efficiently transmit it to the AR terminal.

[0067] First, since the rotation angle and distance are continuous real numbers, the sampling is performed with discrete values. Among the sampled rotation angles, the set of key rotation angles rotating around the x-axis is defined as P, the set of rotation angles rotating around the z-axis is defined as Y. The RMS information at a distance for LOD #1 may be expressed as follows.

[00003]$\begin{array}{cc}{\mathrm{RMS}}_{i,l,k}=\left[\begin{array}{ccc}{\mathrm{rms}}_{1,1}^{i,l,k}& .Math.& {\mathrm{rms}}_{1,.Math.P.Math.}^{i,l,k}\\ .Math.& \ddots & .Math.\\ {\mathrm{rms}}_{.Math.Y.Math.,1}^{i,l,k}& .Math.& {\mathrm{rms}}_{.Math.Y.Math.,.Math.P.Math.}^{i,l,k}\end{array}\right]& \left[\mathrm{Equation}5\right]\end{array}$

[0068] In the above equation, at the distance k, when rotated nth (∈P) about the z-axis and rotated by the mth angle (∈Y) around the x-axis, rms.sub.m,n.sup.i,l,k is the RMS between the image displayed on the screen of the AR terminal when AR content #i renders a chunk of LOD #, and the image displayed on the screen of the AR terminal when AR content #i renders the maximum LOD chunk. The above RMS matrix uses downsampling and singular value decomposition (SVD) to reduce the size of the RMS matrix. In order to reduce the dimension of the RMS matrix, downsampling is performed as shown below, and then decomposed with the SVD.

[00004]$\begin{array}{cc}\mathrm{Down}\left(n^{\mathrm{scale}},{\mathrm{RMS}}_{i,l,k}\right)=U_{i,l,k}{S}_{i,l,k}{V}_{i,l,k}^T& \left[\mathrm{Equation}6\right]\end{array}$$\begin{array}{cc}{U}_{i,l,k}=\left[u_1^{i,l,k}.Math.u_{\frac{.Math.P.Math.}{n^{\mathrm{scale}}}}^{i,l,k}\right]& \left[\mathrm{Equation}7\right]\end{array}$$\begin{array}{cc}{S}_{i,l,k}=\mathrm{diag}\left({\delta }_1^{i,l,k},{\delta }_2^{i,l,k},.Math.,{\delta }_{n_{\max }^{\mathrm{eigen}}}^{i,l,k}\left(\right)\right)& \left[\mathrm{Equation}8\right]\end{array}$$\begin{array}{cc}{V}_{l,k}=\left[v_1^{l,k}.Math.v_{\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}}^{l,k}\right]& \left[\mathrm{Equation}9\right]\end{array}$

[0069] In the above equation, Down is an operator that reduces the RMS matrix in the |P|×|Q| dimension to

[00005]$\frac{.Math.P.Math.}{n^{\mathrm{scale}}}⨯\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}$

by using downsampling, U.sub.i,l,k and V.sub.i,l,k are rectangular orthogonal matrixes that satisfy U.sub.i,l,kU.sub.i,l,k.sup.T=I and V.sub.i,l,kV.sub.i,l,k.sup.T=I, and S.sub.i,l,k is a diagonal matrix composed of the square of the singular values in which the diagonal elements are arranged in descending order. n.sub.max.sup.eigen has a value of min

[00006]$\left(\frac{.Math.P.Math.}{n^{\mathrm{scale}}},\frac{.Math.Y.Math.}{n^{\mathrm{scale}}}\right),$

u.sub.m.sup.i,l,k and v.sub.m.sup.i,l,k are orthogonal eigenvectors associated with a singular value. The RMS matrix reduced by the downsampling is decomposed into S, U, and V matrices through the SVD process. In the matrix decomposed through the SVD, more information of the original RMS matrix is included in the singular values with high values and related vectors.

[0070] In the present disclosure, by using only large singular values and related vectors, the RMS matrix is reconstructed through inverse SVD and upscaling as follows and used for chunk scheduling.

RMS.sub.i,l,k.sup.rec=UP(n.sup.scale,Σ.sub.m=1.sup.eignu.sub.m.sup.i,l,kδ.sub.m.sup.i,l,kv.sub.m.sup.i,l,k.sup.T)  [Equation 10]

[0071] In the above equation, UP is an upscaling operator that increases the size of the matrix by n.sup.scale times, and n.sup.eigen is the number of singular values used to reconstruct the RMS matrix through the upscaling. In the present disclosure, n.sup.eigen and n.sup.scale, which guarantee the R-squared value between the original RMS matrix, RMS.sub.i,l,k and the RMS matrix generated through upscaling, RMS.sub.i,l,k.sup.rec is higher than 0.95, are determined through global search. Therefore, by providing only the n.sup.eigen singular values and related eigenvectors, not the entire RMS matrix in the mobile terminal, the RMS information according to the rotation angle included in P and Y can be transmitted. When a 3D object is rendered and playbacked by rotating at an angle not belonging to P and Y, the RMS value at the adjacent rotation angle included in P and Y is obtained by linear interpolation.

[0072] 2) Chunk Scheduling Process

[0073] In the present disclosure, a low-level chunk including a low-detail mesh is first requested without waiting until all chunks constituting the AR content are received using a progressive mesh. Next, the AR content is rendered and played on the screen as soon as it is transmitted to the AR terminal. After that, a high-level chunk containing the previously transmitted high-detail mesh is requested. Additionally requested high-level chunk is used to progressively enhance the detail of the corresponding AR content as soon as it arrives. In order to determine which level of chunk of which AR content is requested first when multiple AR contents are requested simultaneously, the WRMS defined as follows is defined in the present disclosure as an objective spatial perceptual quality measure for the AR streaming service.

[00007]$\begin{array}{cc}\mathrm{WRMS}\left(\stackrel{.fwdarw.}{d}\right)=\underset{i=1}{\overset{N_{\mathrm{AR}}}{.Math.}}\omega \left(i,\stackrel{.fwdarw.}{d}\right).Math.\mathrm{RM}{S}_i& \left[\mathrm{Equation}11\right]\end{array}$

[0074] In the above equation, {right arrow over (d)} is the distance vector of the AR content (the display scale value of the AR content playbacked on the screen is inversely proportional to the distance between the AR mobile device and the marker), RMS.sub.i is the RMS value of the AR content #i, and ω(i, {right arrow over (d)}) is the scale weight of the AR content #i displayed on the screen. It is inversely proportional to the distance between the mobile AR terminal and the marker, and the sum of the weights is set to 1 (Σ.sub.i=1.sup.N.sup.ARω(i, {right arrow over (d)})=1). In the present disclosure, the next chunk is selected to minimize the above WRMS during the AR streaming service. To this end, the chunk scheduling problem is formalized as follows:

[0075] Problem Definition: Given {right arrow over (ψ)}, {right arrow over (θ)}, and {right arrow over (d)}, determine the AR content #i that maximizes the equation below.

[00008]$\begin{array}{cc}\frac{\omega \left(i,\stackrel{.fwdarw.}{d}\right)({\mathrm{RMS}}_i\left({\theta }_i,{\psi }_i,d_i,l_i^{\mathrm{trans}}\right)-{\mathrm{RMS}}_i\left({\theta }_i,{\psi }_i,d_i,l_i^{\mathrm{trans}}+1\right)}{c_{i,l_i^{\mathrm{trans}}+1}^{\mathrm{byte}}}& \left[\mathrm{Equation}12\right]\end{array}$$\begin{array}{cc}\mathrm{subject}\mathrm{to}{l}_i^{trans}+1<L_i^{\max }& \left[\mathrm{Equation}13\right]\end{array}$$\begin{array}{cc}{T}^{\mathrm{con}}+\frac{c_{i,l_i^{\mathrm{trans}}}^{byte}}{}<T^{\max }& \left[\mathrm{Equation}14\right]\end{array}$

[0076] In the above equation, {right arrow over (l)}.sup.trans=(l.sub.1.sup.trans, l.sub.2.sup.trans, . . . , i.sub.N.sub.AR.sup.trans) is the LOD chunk vector already received by the AR terminal, and l.sub.i.sup.trans denotes the LOD of the AR content #i chunk received so far.

[0077] {right arrow over (θ)}=(θ.sub.1, θ.sub.2, . . . , θ.sub.N.sub.AR) is a rotation angle vector about the x-axis, and θ.sub.i is the angle at which AR content #i is rotated about the x-axis.

[0078] {right arrow over (ψ)}=(ψ.sub.1, ψ.sub.2, . . . , ψ.sub.N.sub.AR) is an angle vector rotated around the z-axis, where ψ.sub.f is the angle at which AR content #i is rotated around the z-axis. {right arrow over (d)}=(d.sub.1, d.sub.2, . . . , d.sub.N.sub.AR) is the distance vector, and d.sub.i is the distance between the mobile AR device and the marker of the AR content #i.

[0079] RMS.sub.i(ψ.sub.i, θ.sub.i, d.sub.i, l.sub.i.sup.trans) is replaced with {tilde over (R)}MS.sub.i(ψ.sub.i, θ.sub.i, d.sub.i, l.sub.i) obtained by calculating by linear interpolation of an adjacent RMS.sub.i,l,k.sup.rec matrix, L.sub.i.sup.max is the maximum LOD of AR content #i, and c.sub.i,j.sup.byte is the chunk size of LOD #j of AR content #i.

[0080] {tilde over (t)}r.sub.tcp is the TCP throughput of the AR terminal, T.sup.con is the time consumed from the start of AR streaming to the present, and T.sup.max is the maximum allowable waiting time, and this value may be set differently by a mobile user. The purpose of the proposed chunk scheduling is to determine the next LOD chunk to minimize WRMS up to T.sup.max.

[0081] Equation 13 means that the next chunk of LOD of the next requested AR content #i must be smaller than the maximum LOD of the AR content, and Equation 14 represents the constraint that the next requested chunk should arrive within T.sup.max. Details of the chunk scheduling process in the AR terminal are as follows.

[0082] In step 100, {right arrow over (l)}.sup.trans is initialized to 0.

[0083] In step 200, c.sub.i,j.sup.byte (for 1≤i≤N.sub.AR and 1≤j≤L.sub.i.sup.max( )) and S, V, and D values are transmitted from the AR server, and the RMS matrix is reconstructed through the S, V, and D.

[0084] In step 300, the AR terminal identifies {right arrow over (θ)}, {right arrow over (ψ)}, {right arrow over (d)}, and {tilde over (t)}r.sub.tcp.

[0085] In step 400, it is checked whether or not the expected download time for the LOD #l.sub.i.sup.trans+1 chunk of the AR content #i exceeds T.sup.max. If one or more chunks excess, the process goes to step 600.

[0086] In step 500, after selecting the AR content #i that maximizes Equation 12, the process goes to step 700.

[0087] In step 600, when receiving the LOD #l.sub.i.sup.trans+1 chunk, the AR content #i that minimizes Equation 15 considering the tradeoff between the WRMS and the transmission delay is selected.

[00009]$\begin{array}{cc}\left(1-\alpha \right)\left(1-\frac{\omega \left(i,\stackrel{.fwdarw.}{d}\right)\left({\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}\right)-{\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}+1\right)\right)}{{.Math.}_{i=1}^{N_{\mathrm{AR}}}\omega \left(i,\stackrel{.fwdarw.}{d}\right)\left({\mathrm{RMS}}_i\left({\psi }_i,{\theta }_i,d_i,l_i^{\mathrm{trans}}\right)\right)}\right)+\alpha .Math.\max \left(\frac{\left(T^{\mathrm{con}}+\frac{c_{i,l_i^{\mathrm{trans}}+1}^{\mathrm{byte}}}{{\mathrm{tcp}}_{}}\right)-T^{\max }}{T^{\max },0}\right).& \left[\mathrm{Equation}15\right]\end{array}$

[0088] In step 700, the next chunk of the selected AR content #i is requested.

[0089] In step 800, when the requested chunk is received, the l.sub.i.sup.trans of {right arrow over (l)}.sup.trans is updated with l.sub.i.sup.trans+1.

[0090] In step 900, steps 300 to 800 are repeatedly performed until all progressive mesh chunks are received.

TABLE-US-00001 TABLE 1 Algorithm: Chunk Scheduling Process Input: {right arrow over (θ)}, {right arrow over (ψ)} , {right arrow over (d)} ,  .sub.tcp, c.sub.i,j.sup.byte , and RMS.sub.i,j,k.sup.rec (for 1 ≤ i ≤ N.sub.AR, 1 ≤ j ≤ Li.sup.max , and k ∈ D)  1: Initialize l.sub.i.sup.trans ← 0 for 1 ≤ i ≤ N.sub.AR  2: While l.sub.i.sup.trans < L.sub.i.sup.max for ∃i  3:  Update {right arrow over (ψ)} , {right arrow over (θ)}, {right arrow over (d)} , through the marker detector  4:  Update  .sub.tcp through the networking monitor  5:  [00010]$\mathrm{If}{T}^{\mathrm{con}}+\frac{c_{t,l_i^{\mathrm{trans}}}^{\mathrm{byte}}}{{\mathrm{tr}}_{\mathrm{tcp}}^{\pm }}<T_{\max }\mathrm{for}\forall i$  6:   Select i that maximizes Eq. 12  7:  Else  8:   Select i that minimizes Eq. 15  9:   Request the chunk of the LOD #l.sub.i.sup.trans + 1   for AR content #i 10: Wait until receiving requesting chunk, then update l.sub.i.sup.trans to l.sub.i.sup.trans + 1 11: End while

[0091] FIG. 5 is an XML structure diagram of a proposed meta file of a system for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0092] Referring to FIG. 5, a meta file structure and format of a system for providing an adaptive AR streaming service according to an embodiment of the present disclosure is disclosed.

[0093] The AR object is composed of multilevel progressive mesh chunks, and its properties are inserted into the meta file. One of the unique properties of the proposed meta file includes RMSE information and bitrate information for multilevel progressive mesh chunks.

[0094] Since the rate distortion information of the meta file is very helpful in improving the quality of a video streaming service in a time-varying wireless network, the proposed meta file has a hierarchical structure as shown in FIG. 5. In the uppermost layer, Marker_ID indicates a detected marker ID. The ArObject layer indicates the name of the AR object corresponding to the marker ID. The ArAdaptationSet layer is composed of several LevelOfDetails including the Distance and ArChunkURL of each LOD chunk. The Distance represents the distance between the mobile AR device and the detected marker. The lower level of the Distance includes RMSE matrix reconstruction information such as key angle, dimension, singular value (S) and associated vectors (U and V). ArChunkURL indicates the URL and size (bytes) of the progressive mesh chunk. Based on the RMSE and the chunk size, the mobile AR device uses HTTP to select the next request chunk to be downloaded from the identified URL.

[0095] FIG. 6 is a configuration diagram of an apparatus 1000 for providing an adaptive AR streaming service according to an embodiment of the present disclosure.

[0096] Referring to FIG. 6, the image-based texture expression motion generation and merging apparatus 1000 according to an embodiment of the present disclosure may be configured to include a processor 1100, a memory 1200, a transceiver 1300, and an input interface device 1400, an output interface device 1500, a storage device 1600 and a bus 1700.

[0097] The apparatus 1000 for providing an adaptive AR streaming service of the present disclosure may include at least one processor 1100 and a memory 1200 that stores the instructions for instructing the at least one processor to perform at least one step.

[0098] The processor 1100 may refer to a central processing unit (CPU), a graphics processing unit (GPU) or a dedicated processor on which methods according to embodiments of the present disclosure are performed.

[0099] Each of the memory 1200 and the storage device 1600 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 1200 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

[0100] Further, the apparatus 1000 for providing an adaptive AR streaming service may include the transceiver 1300 that performs communication through a wireless network.

[0101] Still further, the apparatus 1000 for providing an adaptive AR streaming service may further include the input interface device 1400, the output interface device 1500, the storage device 1600, and the like.

[0102] Still further, each component included in the apparatus 1000 for providing an adaptive AR streaming service may be connected by a bus 1700 to communicate with each other.

[0103] As examples of the device 1000 for providing an adaptive AR streaming service of the present disclosure, there may be a communicable desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone, a smart watch, a smart glass, an e-book reader, a portable multimedia player (PMP), a portable game console, a navigation device, a digital camera, a digital multimedia broadcasting (DMB) player, a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a personal digital assistant (PDA) and the like.

[0104] The exemplary embodiments of the present disclosure may be implemented as program instructions executable by a variety of computers and recorded on a computer readable medium. The computer readable medium may include a program instruction, a data file, a data structure, or a combination thereof. The program instructions recorded on the computer readable medium may be designed and configured specifically for the present disclosure or can be publicly known and available to those who are skilled in the field of computer software.

[0105] Examples of the computer readable medium may include a hardware device such as ROM, RAM, and flash memory, which are specifically configured to store and execute the program instructions. Examples of the program instructions include machine codes made by, for example, a compiler, as well as high-level language codes executable by a computer, using an interpreter. The above exemplary hardware device can be configured to operate as at least one software module in order to perform the embodiments of the present disclosure, and vice versa.

[0106] While the exemplary embodiments of the present disclosure and their advantages have been described in detail, it should be understood that various changes, substitutions and alterations may be made herein without departing from the scope of the present disclosure.