INTEROPERABLE 3D IMAGE CONTENT HANDLING

Abstract

A system and computer-implemented method (s) are described which provide a backward-compatible way of outputting 3D image content by an application. Namely, the application may output the 3D image content as a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data. The application may mutually arrange both views such that a ‘legacy’ 2D display server component for a 2D display omits drawing, or overdraws, the secondary view when compositing the at least two views into a display signal, while a 3D display server component may access the 3D-enabling auxiliary data in the secondary view and process the 2D image data and the 3D-enabling auxiliary data appropriately.

Claims

1. A system for displaying views of an application on a display, wherein the views represent 3D image content for a 3D display, the system comprising: a memory comprising system data representing an operating system and application data representing the application; a processor subsystem configured to communicate with the memory and to execute the operating system and the application; wherein the operating system is configured to provide: a window manager for managing visibility of views generated by applications; one or more display server components, the one or more display server components being specific to a type of display and configured to, based on visibility information obtained from the window manager, composite the views into a display signal for display; wherein the application is configured to provide the 3D image content to the window manager in the form of at least two views which are arranged with respect to each other in accordance with a view configuration, wherein the at least two views comprise a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data, wherein the application is configured to: i) stack the primary view in front of the secondary view to provide a view configuration in which the primary view occludes the secondary view, or ii) provide the primary view to the window manager by indicating a viewport which comprises the 2D image data of the primary view and provide the secondary view to the window manager by arranging the 3D-enabling auxiliary data of the secondary view outside of the viewport, to cause a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal.

2. The system according to claim 1, wherein the application is configured to assign a relative Z-order to the primary view and the secondary view which causes the primary view to be stacked in front of the secondary view.

3. The system according to claim 1, wherein the application is configured to provide a barrier view stacked in between the primary view and the secondary view, wherein the barrier view is opaque and comprises homogenous image data.

4. The system according to claim 1, wherein the one or more display server components comprise a 3D display server component for a 3D display, and wherein the application is configured to signal the 3D display server component that the at least two views represent 3D image content.

5. The system according to claim 4, wherein the 3D display server component provides an API for allowing applications to interface with the 3D display server component, and wherein the application is configured to signal the 3D display server component via the API that the at least two views represent 3D image content, for example by registering an identifier of the at least two views with the 3D display server component via the API.

6. The system according to claim 1, wherein the one or more display server components comprise a 3D display server component, and wherein the 3D display server component is configured to detect that the at least two views represent 3D image content based on metadata of the application.

7. The system according to claim 6, wherein the metadata of the application comprises an identifier of the application.

8. The system according to claim 4, wherein the 3D display server component is configured to composite the at least two views by: arranging the primary view and the secondary view to be simultaneously shown in the display signal in accordance with a stereo display format; or generating one or more further views based on the primary view and the secondary view, and simultaneously arranging the primary view and the one or more further views in the display signal in accordance with a multiview display format.

9. The system according to claim 1, wherein the 3D-enabling auxiliary data is one of the group of: further 2D image data which together with the 2D image data represents a pair of stereo images, or depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer.

10. A display device comprising the system according to claim 1.

11. A computer-implemented method of providing views of an application to a window manager of an operating system, wherein the views represent 3D image content for a 3D display, wherein the operating system is configured to provide: the window manager, wherein the window manager is configured to manage visibility of views generated by applications; one or more display server components, the one or more display server components being specific to a type of display and configured to, based on visibility information obtained from the window manager, composite the views into a display signal for display; the method comprising, by the application, providing the 3D image content to the window manager in the form of at least two views which are arranged with respect to each other in accordance with a view configuration, wherein the at least two views comprise a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data, wherein said providing of the 3D image content by the application comprises: i) stacking the primary view in front of the secondary view to provide a view configuration in which the primary view occludes the secondary view, or ii) providing the primary view to the window manager by indicating a viewport which comprises the 2D image data of the primary view and provide the secondary view to the window manager by arranging the 3D-enabling auxiliary data of the secondary view outside of the viewport, to cause a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal.

12. A or non-transitory computer-readable medium comprising a computer program representing an application, the computer program comprising instructions for causing a processor system to perform the method according to claim 11.

13. A computer-implemented method of compositing views of an application executed on an operating system, wherein the views represent 3D image content for a 3D display, wherein the operating system is configured to provide: a window manager for managing visibility of views generated by applications; a 3D display server component for a 3D display, the 3D display server component being configured to, based on visibility information obtained from the window manager, composite the views into a display signal for the 3D display; wherein the application is configured to provide the 3D image content to the window manager in the form of at least two views which are arranged with respect to each other in accordance with a view configuration, wherein the at least two views comprise a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data which is indicative of depth of the 2D image data, wherein the application is configured to: i) stack the primary view in front of the secondary view to provide a view configuration in which the primary view occludes the secondary view, or ii) provide the primary view to the window manager by indicating a viewport which comprises the 2D image data of the primary view and provide the secondary view to the window manager by arranging the 3D-enabling auxiliary data of the secondary view outside of the viewport, to cause a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal; the method comprising, by the 3D display server component: determining that the at least two views represent 3D image content on the basis of signaling received from the application, or metadata of the application; and processing the primary view and the secondary view to obtain the display signal for the 3D display.

14. A transitory or non-transitory computer-readable medium comprising a computer program representing a 3D display server component, the computer program comprising instructions for causing a processor system to perform the method according to claim 13.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0051] These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,

[0052] FIG. 1 shows a schematic overview of interactions between an application, a window manager and several display server components, in which the application provides 3D image content to the window manager in the form of a primary view comprising 2D image data and a secondary view comprising 3D-enabling auxiliary data, and the display server components are each configured to composite the views of applications into a display signal for display on a respective type of display based on visibility information obtained from the window manager;

[0053] FIG. 2A illustrates a result of the application stacking the primary view in front of the secondary view to cause a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the views into the display signal;

[0054] FIG. 2B shows the application indicating a viewport to the window manager which comprises the 2D image data of the primary view, while the 3D-enabling auxiliary data of the secondary view is located outside of the viewport;

[0055] FIG. 3 shows a specific embodiment for the Android Operating System;

[0056] FIG. 4 shows a system for displaying views of an application on a display; and

[0057] FIG. 5 shows a computer readable medium comprising instruction data.

[0058] It should be noted that items which have the same reference numbers in different Figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.

LIST OF REFERENCE NUMBERS

[0059] The following list of reference numbers is provided for facilitating the interpretation of the drawings and shall not be construed as limiting the claims. [0060] 080 (3D) display [0061] 100 system for displaying views of application [0062] 120 processor subsystem [0063] 122 data communication [0064] 140 memory [0065] 142 data communication [0066] 160 bulk storage [0067] 162 data communication [0068] 180 display output [0069] 182 display data [0070] 200 application [0071] 210 buffer [0072] 212 buffer [0073] 214 buffer [0074] 220 window manager [0075] 240 display server [0076] 252 2D display server component [0077] 254 3D (multiview) display server component [0078] 256 3D (stereo) display server component [0079] 260 Android BSP [0080] 262 Android WindowManager [0081] 264 Android SurfaceFlinger [0082] 280 Service Process [0083] 282 Extension (Hook) [0084] 284 Service API [0085] 300 2D image data [0086] 310 3D-enabling auxiliary data [0087] 400 primary view, 2D view [0088] 410 secondary view, 3D-enabling auxiliary view [0089] 420 Z-order [0090] 430 viewport [0091] 440 display signal for 2D display [0092] 442 display signal for 3D (multiview) display [0093] 444 display signal for 3D (stereo) display [0094] 500 computer readable medium [0095] 510 instruction data

DESCRIPTION OF THE INVENTION

[0096] FIG. 1 shows a schematic overview of software components of or running on an operating system, namely an application 200, a window manager 220 and a display server 240, which is shown to comprise, by way of example, a 2D display server component 252 (labeled ‘2D Composition’ in FIG. 1) for a 2D display, a 3D display server component 254 (labeled ‘2D+3D aux Composition’) for a multiview 3D display, and another 3D display server component 256 (labeled ‘2D+3D aux Composition+Stereo out’) for a stereo 3D display. The view composition of each of these display server components 252-256 will be explained further onwards with reference to FIG. 2A and continued reference to FIG. 1.

[0097] As can be seen in FIG. 1, the application 200 may receive or retrieve 2D image data 300 and accompanying 3D-enabling auxiliary data 310, for example from a bulk storage device such as a solid-state drive or hard disk (not shown in FIG. 1), or in a streaming manner from a local or wide area network (also not shown in FIG. 1). In a specific example, the 2D image data 300 and the 3D-enabling auxiliary data 310 may together represent a 3D image or a 3D video frame of a 3D video (such as a movie or short video clip). In yet another example, the application may generate the 2D image data 300 and the 3D-enabling auxiliary data 310, for example by computer rendering techniques providing stereo images or an image and accompanying depth map as output. In yet another example, the application 200 may receive or retrieve the 2D image data 300 and generate the 3D-enabling auxiliary data, for example using depth estimation techniques which are known per se in the part.

[0098] The application 200 may then generate a 2D view 400 which comprises the 2D image data 300. The 2D view 400 is also referred to as ‘primary view’. Here, the term ‘comprises’ refers to the 2D view 400 normally showing the 2D image data 300, albeit subject to visibility limitations of the 2D view 400 which may, for example, be imposed by the window manager 220. The application 200 may further generate a 3D-enabling auxiliary view 410 which comprises the 3D-enabling auxiliary data. The 3D-enabling auxiliary view 410 is also referred to as ‘secondary view’. The data of both views 400, 410 may, for example, be stored in respective buffers 210, 212, for example as buffer objects, or in a same buffer as separate buffer objects (not shown in FIG. 1). The application 200 may provide both views 400, 410 to the window manager 220, for example by using addView( ) of the android.view.WindowManager class. The window manager may generally manage visibility of the views of applications. For example, if there are other active applications, it may determine whether the views 400, 410 are principally visible, or whether they are principally not visible, for example, by being occluded by another application's view(s) or by a window of the application 200 which comprises the views 400, 410 being ‘minimized’.

[0099] The primary view 400 and the secondary view 410 are mutually arranged by the application in accordance with a view configuration which causes a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal. There are various ways to define or provide such a view configuration, as e.g. elucidated with reference to FIGS. 2A and 2B.

[0100] FIG. 2A illustrates a result of the application stacking the primary view 400 in front of the secondary view 410, thereby occluding the secondary view 410. It is noted that FIG. 2A shows, for sake of illustration, a slanted perspective so as to show the secondary view 410 underlying the primary view 400. The display server component, however, considers the primary view 400 to entirely, or at least substantially entirely, occlude the secondary view 410. For example, the secondary view 410 may be assigned a Z-order by the application which puts it behind the primary view 400. At the same time, the location and size of the secondary view 410 may be chosen such that it is (substantially) entirely occluded by the primary view 400. For example, both views 400, 410 may be assigned a same location and size, e.g., in a window of the application (not explicitly shown in FIG. 2A). The stacked views 400, 410 may then cause a ‘legacy’ display server component, e.g., a display server component which is not able to determine that the stacked views 400, 410 represent 3D image content, to overdraw the secondary view 410 with the primary view 400 when compositing the views into the display signal, thereby occluding the secondary view 410.

[0101] This concept may also be referred to as ‘explicit overdraw’ in which an output image is updated multiple times. A black or otherwise opaque and homogenous barrier view may be inserted in between the primary view 400 and the secondary view 410 in case the primary view 400 is (semi) transparent. Here, “inserted in between” refers to the barrier view having a Z-order placing it behind the primary view 400 but before the secondary view 410, with the barrier view preferably having a same location and size as the primary view 400 and/or the secondary view 410. Alternatively, if the display server component is able to determine that the secondary view 410 is occluded, the display server component may also entirely omit drawing the secondary view 410.

[0102] It is noted that to cause the overdrawing of ‘legacy’ display server components, the views 400, 410 generated by the application may be of a type that can only be composed by the operating system in order to bypass app composition optimization. For example, in Android, this may be achieved by the views 400, 410 being ‘SurfaceView’ type of views.

[0103] With continued reference to FIGS. 1 and 2A, a 3D display server component 254, 256 may be configured to determine that the stacked views 400, 410 of FIG. 2A represent 3D image content. As also later elucidated with reference to FIG. 3, this may be determined for example on the basis of signaling received from the application 200, or metadata of the application 200. Accordingly, the 3D display server component 254, 256 may process the primary view 400 and the secondary view 410 to obtain the display signal for the 3D display. Various ways of processing and compositing are known per se in the art of 3D processing and 3D display.

[0104] FIG. 1 shows two non-limiting examples. According to a first example, the 3D display server component 254 may be configured to generate a display signal 442 for a multiview display by using a view-rendering or view-synthesis technique to generate, from the 2D image data and using the 3D-enabling auxiliary data, other viewpoints of the scene shown by the 2D image data, namely in the form of other 2D images labeled, by way of example, ‘X’, ‘Y’ and ‘Z’ in FIG. 1. Such viewpoints are often also referred to as ‘views’ (hence the name ‘view-rendering’ and ‘view-synthesis’) but are not to be confused with the application views 400, 410. The type of view-rendering or view-synthesis may depend on the type of 3D-enabling auxiliary data 310, but normally uses depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer. If the 3D-enabling auxiliary data 310 rather represents further 2D image data which together with the 2D image data represents a pair of stereo images, such depth-related data may be first generated from the pair of stereo images, e.g., using disparity estimation techniques as known per se.

[0105] The 3D display server component 254 may then create a mosaic-like composition of the 2D image data of the primary view 400, labeled ‘2D’ in FIG. 1, and the 2D image data of the other view(point)s, being in this example the three further view(point)s X, Y and Z. The display signal 442 may then be provided as output image signal to a 3D multiview display which may then render the scene in 3D using the image data of the different view(point)s.

[0106] It is noted that also various other known types of multiview compositions may be used instead of a spatial mosaic-like composition. For example, the views may be spatially and/or temporally interleaved. Also other data may be provided in the display signal in addition to or instead of the viewpoints. For example, the 3D-enabling auxiliary data itself may be provided in the display signal, and/or transparency information and/or occlusion information. Another example is that the display signal may comprise a point cloud or a 3D mesh, etc., which are generated from the at least two views. It is noted that in general, the display signal may be provided to the display over a single but also over multiple independent data channels and/or and cables, e.g. via dual DisplayPort.

[0107] According to a second example, the 3D display server component 256 may be configured to generate a display signal 444 for a stereo display. This view composition may take different forms depending on the type of 3D-enabling auxiliary data 310. For example, the 3D-enabling auxiliary data 310 represents further 2D image data which together with the 2D image data represents a pair of stereo images. In this case, if the 2D image data 300 represents a left image and the further 2D image data 310 represents a right image, the 3D display server component 256 may composite them into a side-by-side formatted display signal 444. It is noted that also various other known types of spatial and/or temporal stereo view compositions may be used instead. If, however, the 3D-enabling auxiliary data 310 represents depth-related data indicative of a distance of objects shown in the 2D image data to a camera or viewer, the further 2D image data (e.g., the right image) may be generated using the aforementioned view-rendering or view-synthesis techniques.

[0108] FIG. 2B shows a further example of the application mutually arranging the primary view 400 and the secondary 410 in accordance with a view configuration which causes a 2D display server component for a 2D display to omit drawing, or overdraw, the secondary view when compositing the at least two views into the display signal. Namely, in this example, the application indicates a viewport 430 to the window manager which comprises the 2D image data of the primary view 400. For example, the viewport 430 may be defined with respect to a buffer 214 (which in general may be a ‘buffer object’ referring to the buffer being a logical buffer rather than a specific physical buffer). As also shown in FIG. 2B, the 3D-enabling auxiliary data of the secondary view 410 may then be located outside of the viewport 430 in the buffer 214, causing a ‘legacy’ display server component to omit drawing the secondary view 410. Alternatively, the 3D-enabling auxiliary data of the secondary view 410 may be located outside of the viewport 430 in a different manner, e.g., in a different buffer.

[0109] FIG. 3 shows a specific embodiment for the Android Operating System. Again, an application 200 is shown to retrieve or receive 2D image data 300 and 3D-enabling auxiliary data 310, and to provide a primary view 400 comprising the 2D image data and a secondary view 410 comprising the 3D-enabling auxiliary data. Although FIG. 3 shows the primary view 400 being stacked on top the secondary view 410, both views may also be differently mutually arranged so as to cause a ‘legacy’ display server component to overdraw, or omit drawing, the secondary view 410. This also applies to the views shown in FIG. 1.

[0110] In the example of FIG. 3, the application 200 may provide the secondary view 410 as full-screen and arrange the primary view 400 on top of the secondary view 410.

[0111] FIG. 3 further shows a window manager 262 and a SurfaceFlinger 264 which may be part of an Android Board Support Package (BSP) 260. The SurfaceFlinger 264 may embody at least part of a 3D display server component by processing the application views and to perform, or at least to enable view composition for a 3D display.

[0112] There are various ways to enable the 3D display server component 264 to determine that the stacked views 400, 410 of the application 200 represent 3D image content. In the example of FIG. 3, the application 200 may register the stacked views 400, 410 to represent 3D image content with a service process 280 via a service API 284, with the service process 280 being able to communicate with the 3D display server component 264 via an extension (‘hook’) 282. Accordingly, the 3D display server component may be informed that the stacked views 400, 410 of the application represent 3D image content, and may process the views accordingly. The service process 280 may be purposefully provided to enable a 3D display server component to determine that the views of an application represent 3D image content.

[0113] Although described with reference to Android, the above measures may be applied to various other types of operating systems by a person skilled in the art of application development for the particular operating system. For example, for *nix-based operating systems (Linux, BSD, Unix, etc.), a wide variety of window managers exist (see https://wiki.archlinux.org/index.php/window_manager) as well as various customizable display servers (e.g., X11, Wayland-based, Mir, DirectFb). For MacOS or iOS (Quartz), the Quartz Window Services may provide at least part of the functionality of the window server as described, and the Quartz Display Services/Quartz Composition Services/XQuartz may configured to provide at least part of the functionality of the display server of display server components as described. For Microsoft Windows, the Desktop Window Manager may provide at least part of the functionality of the window server as described, and the GDI Graphics Device Interface may be configured to provide at least part of the functionality of the display server or display server components as described.

[0114] FIG. 4 shows a system 100 for displaying views of an application on a display 080. Such a system 100 may be configured to run the operating system and application as described within this specification, for example with reference to FIGS. 1-3. The system 100 may comprise a memory 140 which may comprise system data representing the operating system and application data representing the application. The system 100 may further comprise a processor subsystem 120 configured to communicate with the memory via data communication 142 and to execute the operating system and the application. By way of example, the system 100 is further shown to comprise a display output for outputting display data 182 to a display 080 connected to the system 100. The display data may represent visualized output of the processor system 120, such as the display signal generated by the display server component, which visualized output may be provided to the display output 180 via data communication 122. By way of example, the system 100 is further shown to comprise a bulk storage 160 which may store the 2D image data and the 3D-enabling auxiliary data, which data may be accessed by the processor subsystem via data communication 162.

[0115] In general, the system may be embodied in or as a separate device, e.g., in or as a set-top box, personal computer, gaming console or similar device that is connectable to a (3D) display. Alternatively, the system may be embodied in or as a display device which comprises the (3D) display, e.g., in or as a smartphone, tablet device, television, display, monitor, etc. In general, the system may be implemented by a device or apparatus. The device or apparatus may comprise one or more (micro) processors which execute appropriate software. Software implementing the functionality of the function(s) may have been downloaded and/or stored in a corresponding memory or memories, e.g., in volatile memory such as RAM or in non-volatile memory such as Flash. Alternatively, the function(s) of the system may be implemented in the device or apparatus in the form of programmable logic, e.g., as a Field-Programmable Gate Array (FPGA), or as an Application-Specific Integrated Circuit (ASIC), or as any other type of circuit or combination of circuits. Any of the software components described in this specification may be represented by instructions for a computer, e.g., executable code, which may be stored on a computer readable medium 500, e.g., in the form of a series 510 of machine readable physical marks and/or as a series of elements having different electrical, e.g., magnetic, or optical properties or values. The executable code may be stored in a transitory or non-transitory manner. Examples of computer readable mediums include memory devices, optical storage devices, online software, etc. FIG. 5 shows an optical disc 500. For example, the computer readable medium 500 may store an application as described in this specification, and/or a display server component, possibly as part of an operating system.

[0116] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments.

[0117] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of” when preceding a list or group of elements represent a selection of all or of any subset of elements from the list or group. For example, the expression, “at least one of A, B, and C” should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.