SYSTEM AND METHOD FOR RENDERING PERSPECTIVE ADJUSTED VIEWS OF A VIRTUAL OBJECT IN A REAL WORLD ENVIRONMENT

Abstract

A method for rendering perspective adjusted views of a virtual object in a real world environment is provided. A registration code is generated for a first device and includes a static portion for device identification and a dynamic portion for a location and orientation of the first device. The dynamic portion of the registration code changes based on time passage and movement of the first device. A distance and orientation of the first device is determined with respect to a second device based on a location and orientation of the second device at a particular time and the registration code, which is captured by the second device at the particular time. The second device captures the registration code via the first device or a different device. A display of a virtual object is perspective adjusted based on the distance and orientation of the devices.

Claims

1. A method for rendering perspective adjusted views of a virtual object in a real world environment, comprising: maintaining virtual objects stored in a database: generating for a first device, a registration code comprising a static portion for device identification and a dynamic portion for a location and orientation of the first device; changing the dynamic portion of the registration code based on at least one of time passage and movement of the first device; determining a distance and orientation of the first device with respect to a second device based on a location and orientation of the second device at a particular time and further based on the registration code, which is captured by the second device at the particular time, wherein the second device captures the registration code via the first device or a different device; accessing at least one of the virtual objects from the database; and adjusting a display of the virtual objects based on the distance and orientation of the devices.

2. A method according to claim 1, further comprising: receiving instructions from at least one of the first device and the second device to perform an action with respect to the virtual object.

3. A method according to claim 1, wherein the first and second devices each comprise one of a mobile telephone, smart watch, and tablet.

4. A method according to claim 1, further comprising: calculating a size of the registration code captured by the second device; and determining the distance of the second device from the first device based on the size of the registration code representation.

5. A method according to claim 1, wherein the registration code comprises multiple black modules against a white background and a portion of the black modules are static and a portion of the black modules are dynamic.

6. A method according to claim 1, wherein the adjusted display comprises a view of the virtual object based on the location of the first or second device with respect to the virtual object and the other of the second or first device.

7. A method according to claim 1, further comprising: decoding data via alphabetic characters, text, numbers, or URLs for the registration code.

8. A method according to claim 1, further comprising: identifying a user of one of the first device or the second device; and providing a picture of the identified user on an other of the second device or the first device.

9. A method according to claim 1, further comprising: determining the orientation of the first and second devices based on one or more of pitch, roll, yaw and location coordinates comprising x, y or z coordinates.

10. A method according to claim 1, further comprising: changing the display of the virtual object upon movement of at least one of the first device and the second device.

11. A non-transitory computer readable storage medium storing code for executing on a computer system to perform the method comprising: maintaining virtual objects stored in a database: generating for a first device, a registration code comprising a static portion for device identification and a dynamic portion for a location and orientation of the first device; changing the dynamic portion of the registration code based on at least one of time passage and movement of the first device; determining a distance and orientation of the first device with respect to a second device based on a location and orientation of the second device at a particular time and further based on the registration code, which is captured by the second device at the particular time, wherein the second device captures the registration code via the first device or a different device; accessing at least one of the virtual objects from the database; and adjusting a display of the virtual objects based on the distance and orientation of the devices.

12. A non-transitory computer readable storage medium according to claim 11, further comprising: receiving instructions from at least one of the first device and the second device to perform an action with respect to the virtual object.

13. A non-transitory computer readable storage medium according to claim 11, wherein the first and second devices each comprise one of a mobile telephone, smart watch, and tablet.

14. A non-transitory computer readable storage medium according to claim 11, further comprising: calculating a size of the registration code captured by the second device; and determining the distance of the second device from the first device based on the size of the registration code representation.

15. A non-transitory computer readable storage medium according to claim 11, wherein the registration code comprises multiple black modules against a white background and a portion of the black modules are static and a portion of the black modules are dynamic.

16. A non-transitory computer readable storage medium according to claim 11, wherein the adjusted display comprises a view of the virtual object based on the location of the first or second device with respect to the virtual object and the other of the second or first device.

17. A non-transitory computer readable storage medium according to claim 11, further comprising: decoding data via alphabetic characters, text, numbers, or URLs for the registration code.

18. A non-transitory computer readable storage medium according to claim 11, further comprising: identifying a user of one of the first device or the second device; and providing a picture of the identified user on an other of the second device or the first device.

19. A non-transitory computer readable storage medium according to claim 11, further comprising: determining the orientation of the first and second devices based on one or more of pitch, roll, yaw and location coordinates comprising x, y or z coordinates.

20. A non-transitory computer readable storage medium according to claim 11, further comprising: changing the display of the virtual object upon movement of at least one of the first device and the second device.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a functional block diagram showing a system for rendering perspective adjusted views of a virtual object in a real world environment, in accordance with one embodiment.

[0012] FIG. 2 is a process flow diagram showing a method for rendering perspective adjusted views of a virtual object in a real world environment, in accordance with one embodiment.

[0013] FIG. 3 is a block diagram showing, by way of example, a registration code 40.

[0014] FIG. 4 is a block diagram showing, by way of example, a floor plan of a real world environment with a virtual object.

[0015] FIG. 5 is a block diagram showing, by way of example, a perspective view of a virtual object in the real world environment of FIG. 4 from one user perspective.

[0016] FIG. 6 is a block diagram showing, by way of example, a different perspective view of a virtual object in the real world environment of FIG. 4 from a different user perspective.

DETAILED DESCRIPTION

[0017] A dynamically changing registration code provides information to synchronize mobile devices and provide complementary views of virtual objects in a real world setting. FIG. 1 is a functional block diagram showing a system 10 for rendering perspective adjusted views of a virtual object in a real world environment, in accordance with one embodiment. The system 10 can operate via a cloud computing environment 11, which allows end users to access and utilize remotely-stored applications 13 without requiring the users to install software or personal data. Instead, clients receive cloud-based software 13 and stored data. Each of the users can operate a mobile computing device 17a-c, such as a mobile telephone, tablet, or smart watch to access the applications 13 and data 15, 16 stored on remote servers 12 and databases 14. Other types of mobile computing devices are possible. At a minimum, each mobile computing device should include a display screen, user interface, camera, and wireless communication capabilities, such as WiFi, cellular, LTE, and Bluetooth. Wired communications can also be used.

[0018] The mobile devices 17a-c each execute an application program (app) 13, which can be implemented through software, firmware, or hardware for execution locally on the mobile device. The app 13 can be obtained from a server 12 located in the cloud 11. The app 13 generates a registration code 16 for the mobile device 17a-c on which the app is executed. The registration code 16 can include registration information, such as an identification of the mobile device, a location and orientation of the mobile device, and time. Other types of information are possible. In one example, the registration code 16 can appear similar to a matrix barcode, such as a QR code, and can be represented as multiple black squares arranged on a white background in a shape of a square. However, other shapes and types of the registration code are possible.

[0019] As the mobile device 17a-c moves and as time passes, the registration code 16 changes to reflect the movement and time passage. For instance, the black squares can move positions on the white background to reflect the information change and provide the new location and orientation of the device, and time. To synchronize with another mobile device 17a-c for perspective view cooperation with respect to virtual objects, the mobile device 17a-c can display the registration code 16 for scanning by the other mobile devices 17a-c. Subsequently, the app 13 determines a distance and orientation between the device 17a-c and the other mobile device 17a-c to perform synchronization of the mobile devices after a representation of the registration code 16 is captured on the other device. In a further embodiment, the synchronization and provision of the devices can occur via peer-to-peer computing of the mobile devices, rather than relying on a client-server relationship. Once synchronized, virtual objects 15 can be obtained from a database 14 interconnected to one or more servers 12 and displayed on the screens of the synchronized mobile devices 17a-c for perspective adjusted cooperation views.

[0020] The perspective adjusted cooperation views allow users of the mobile devices 17a-c to view the virtual objects 15 from a real world perspective in a real world environment. For example, a user of one mobile device views a front of a virtual car displayed on a screen of the device, which appears to be parked on the street adjacent to a sidewalk in the real world. Another user operating another mobile device is located to the right of the user, on the real world sidewalk, and views the driver side of the virtual car, as if the virtual car was real. The respective perspectives of the users are described in further detail below with respect to FIGS. 3-6.

[0021] To initiate perspective adjusted cooperation between at least two mobile devices, registration and synchronization must occur to orient the devices. FIG. 2 is a process flow diagram showing a method for rendering perspective adjusted views of a virtual object in a real world environment, in accordance with one embodiment. When an initiating user opens an app to initiate an augmented reality game or adventure, a registration code is generated and provided (block 21) to a mobile device of the user, as an initiating device. The registration code encodes information regarding the initiating device of the initiating user, such as adventure or game identification, device identification, location, including coordinate positions, orientation, and time. However, other types of information are possible. FIG. 3 is a block diagram showing, by way of example, a registration code 40. The registration code 40 can include a plurality of black and white shapes or modules 41, such as squares, which encode data and together form a larger shape, such as a square. However, other shapes are possible. The data can include augmented reality session identification, mobile device identification, user identification, location, orientation, and time. Other types of data are possible. The data can be encoded using alphabetic characters, text, numbers, and URLs. At least a portion of the registration code 40 can change to reflect changes in the data, such as user, session, or device location and time. Other parts of the registration code 40 can represent static information, such as experience identification, mobile device identification, and user identification, and the modules can remain in the same position. For example, an outer border of a registration code 40 can represent the static information and remain unchanged, while the modules inside the outer static border change constantly based on the changes in time, location, and orientation of the user. Alternatively, other arears of the registration code and represent static and dynamic modules.

[0022] Returning to the discussion with respect to FIG. 2, the registration code can be displayed on a screen of the user's initiating device to allow other users to join the initiating user in an augmented reality session. Alternatively, the registration code can be displayed on a larger display, such as a television, for capturing by mobile devices of multiple other users at a single time. The other users can capture the registration code by scanning the code with their respective mobile devices, as receiving devices. Upon scanning, a representation of the captured registration code is received (block 22) on the receiving device and registration of the mobile device is performed (block 23). In one embodiment, a server receives the registration code for processing; however, in a peer-to-peer environment, the captured registration code remains on the receiving device for processing. During registration, information from each receiving device that scanned the registration code is obtained and recorded for use throughout the augmented reality game or experience to determine (block 24) a distance and orientation of that receiving device with respect to the initiating user's mobile device. The information obtained from the receiving devices can include device identification, location, and orientation, and can be used with the data from the registration code of the initiating device to determine at least one of distance and orientation between each receiving device and the initiating device.

[0023] The orientation of the receiving device can be determined from information, such as pitch, roll, yaw and location coordinates, x, y and z, obtained from sensors, including orientation sensors and accelerometers, in the mobile device. In one embodiment, the information can be retrieved from a six-axis gyroscope located in the mobile device. However, other configurations and sensors can be used to obtain the information. Meanwhile, a distance of the receiving device from the initiating device can be determined via computer vision analysis based on a size and proportion of the registration code representation captured. For example, smaller representations of the registration code are associated with longer distances between the mobile devices, while larger representations are associated with shorter distances. Alternatively, or in addition, the distance can be determined based on the location or coordinate information obtained from the receiving device and the registration code from the initiating device.

[0024] The information obtained from the receiving device is used with the information provided by the registration code for the initiating device to synchronize (block 25) the devices. Specifically, synchronization utilizes the locations and orientations of the initiating and receiving devices to ensure that the positions of the synched devices are coordinated in space over time. Upon synchronization, the information provided by the receiving device and information for the initiating device can be stored in a file and utilized to track movement of the devices and determine orientation and distance from each other. Subsequently, during the augmented reality game or experience, movement of the synched devices are monitored (block 26) with respect to one another and each device continuously updates the server with its location and orientation. The initiating device, including position information, is used as a frame of reference for the other receiving mobile devices, which are updated relative to the initiating device.

[0025] Virtual objects are displayed (block 27) on a screen of at least one of the synched mobile devices, depending on a position and orientation of the users and mobile devices. The location and orientation information of each mobile device is translated into a correct visual representation relative to a player position for that device. Thus, each user's view of the virtual object depends on her location with respect to the virtual object, as further described below with reference to FIGS. 4-6. Based on a distance to the virtual object, the user can perform an action on or with respect to the virtual object depending on rules of the particular game or experience. Monitoring movement (block 26) of the synched devices and presenting virtual objects (block 27) occurs until the experience is terminated (block 28) by at least one of the users.

[0026] When the users of the synched devices are in a common viewing range, the users' perspectives of a virtual object are adjusted to ensure respective views of the virtual object based on the location and orientation of each user with respect to the virtual object. FIG. 4 is a block diagram showing, by way of example, a floor plan 30 of a real world environment 36 with a virtual object 35. The floor plan 30 is a representation of a real world conference room 36 in which a real conference table 33 and chairs 34 are positioned in a center of the room 36. Two users 31, 32 are located in the room 36, one in a bottom right corner 32 and the other in a top left corner 31, diagonally across from the user 32 in the bottom right corner. The users 31, 32 are oriented to face one another. Each user is holding a mobile computing device, such as a mobile phone 37, 38 on which an augmented reality app is running. The app accesses the camera on each mobile device 37, 38 to provide the corresponding user 31, 32 with a view of the conference room. A virtual object 35, in the shape of a tea cup, is displayed via the mobile devices 37, 38 of the users 31, 32, as being placed on the real world conference room table 33 with the handle 39 of the tea cup facing the user 32 in the bottom right hand corner of the conference room 36.

[0027] The mobile devices 37, 38 of the users 31, 32 are synched and each user 31, 32 receives a perspective corrected view of the virtual tea cup 35, which is representative of the view each user 31, 32 would have if the tea cup was real. FIG. 5 is a block diagram showing, by way of example, a perspective view 50 of a virtual object 35 in the real world environment of FIG. 4. The user's 31 position has been rotated to the bottom of the figure to provide a reader with the same view as the user. Thus, the user 31 views the real world conference table 33 with the other user 32 at the opposite end of the table 33, near the top of the figure. The user 31 views a chair 34 to a left of the other user 32, which is actually located on a right side of the other user 32. The virtual tea cup 35 sits in a center of the conference table 35 with the handle 39 facing the user 31. Thus, the user 31 is able to view the virtual handle 39 of the tea cup 35.

[0028] The other user 32 has a view opposite the user 31. FIG. 6 is a block diagram showing, by way of example, a different perspective view 60 of a virtual object 35 in the real world environment of FIG. 4. The user's 32 position has been rotated to the bottom of the figure to provide a reader with the same view as the user. The perspective view 60 of the user 32 includes the conference room table 33 with a real world chair 34 to a right of the user 32. With respect to the virtual object 35 on the real world conference table 33, the user 32 sees the opposite side of the tea cup 35 than the other user 31. Thus, since the handle (not shown) of the tea cup 35 faces the other user 31, the user 32 cannot see the handle.

[0029] Providing accurate perspective views of virtual objects in a real world environment can be used in augmented reality games, experiences, activities, and simulations. For example, an augmented reality game can include a list of virtual objects which each user participant must find and capture within a real world environment, within a predetermined time. A first player can receive the name of an object, such as a ladybug, which the second user must find and capture. Meanwhile, the second user receives the name of an object, such as a spider, which the first user must find and capture. The two players can collaboratively find the virtual objects within the real world environment. Depending on where the players are looking, the players can see the same or different virtual objects. For instance, the second player may see the spider hanging above the first player's head and tell the first player to look up. Once the first player looks up and sees the spider, the spider can be captured. The views of the spider can be different because of the players' different positions. For instance, the first player looks up and sees the underbelly of the spider, while the second player sees a front of the spider since the second player is positioned across from the spider.

[0030] In a further embodiment, virtual objects can be stacked on top of one another. For instance, a virtual book shelf or other furniture can be built for placement in a real world room to provide a simulated view, such as for a potential remodel. A user can then place virtual books on the shelf and see how the furniture would look in in the room from all angles by walking around the virtual piece of furniture.

[0031] In yet a further embodiment, perspective adjusted virtual views can be used to simulate trying on clothes. A user may want to see whether she would look good in a maxi dress and select to try on a virtual maxi dress. The user can then turn around in front of the camera on her mobile device to see different angles and sides of the virtual dress on.

[0032] Other examples for perspective adjusted virtual views, include interior design, such as placing virtual pillows on a virtual couch, putting a virtual puzzle together, and collecting virtual objects. Once collected, a user can manipulate the object in space, such as by moving the virtual object, adding to the virtual object, or altering the object.

[0033] Facial recognition can be used with the perspective adjusted virtual views to identify players, set preferences, and customize experiences, such as by displaying virtual objects that an identified player likes.

[0034] While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.