The present invention discloses systems and methods for both viewing and interacting with a virtual reality (VR), an augmented reality (AR) or a mixed reality (MR). More specifically, the systems and methods allow the user to interact with aspects of such realities including virtual items presented in such realities or within such environments by manipulating a control device that has an inside-out camera mounted on-board. The apparatus or system uses two distinct representations including a reduced representation in determining the pose of the control device and uses these representations to compute an interactive pose portion of the control device to be used for interacting with the virtual item. The reduced representation is consonant with a constrained motion of the control device.

A user's eyes and if desired head is tracked as the user's gaze follows a moving object on a display. Motion blur of the moving object is keyed to the eye/head tracking. Motion blur of other objects in the frame also may be keyed to the eye/head tracking.

A simultaneous location and mapping (SLAM)-enabled video game system, a user device of the video game system, and a computer-readable storage medium of the user device are disclosed. Generally, the video game system includes a video game console, a plurality of thermal beacons, and a user device communicatively coupled with the video game console. The user device includes a thermopile array, a processor, and a memory. The user device may receive thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile array. The user device may determine, based on the thermal data, its location in 3D space, and then transmit that location to the video game system.

A controller for a computer entertainment system including a control box (104) for executing and running a computer game and causing game footage to be displayed on a screen (510—FIG. 3), said control box (104) being configured to receive control signals representative of user manipulation of said game and dynamically adjust said game footage accordingly; said controller comprising a hand-held gamepad (102) and an inertial sensor (106), separate from and configured for data communication with, said gamepad (102); said hand-held gamepad (102) comprising at least one control input device and including a processor (114—FIG. 5) for generating control signals for manipulating a game running on said control box (104) in response to user manipulation of said at least one control input device and causing said control signals to be transmitted to said control box (104); said inertial sensor (106) being configured to be mounted in or on a user-worn garment such that user movement causes corresponding movement of said inertial sensor (106); said processor (114—FIG. 5) being further configured to receive signals from said inertial sensor (106) representative of user movement, convert said signals into control signals for manipulating said computer game, said control signals being of the same format as control signals generated in response to user manipulation of a respective control input device, and cause said control signals to be transmitted to said control box (104).

A controller for a computer entertainment system including a control box (104) for executing and running a computer game and causing game footage to be displayed on a screen (510—FIG. 3), said control box (104) being configured to receive control signals representative of user manipulation of said game and dynamically adjust said game footage accordingly; said controller comprising a hand-held gamepad (102) and an inertial sensor (106), separate from and configured for data communication with, said gamepad (102); said hand-held gamepad (102) comprising at least one control input device and including a processor (114—FIG. 5) for generating control signals for manipulating a game running on said control box (104) in response to user manipulation of said at least one control input device and causing said control signals to be transmitted to said control box (104); said inertial sensor (106) being configured to be mounted in or on a user-worn garment such that user movement causes corresponding movement of said inertial sensor (106); said processor (114—FIG. 5) being further configured to receive signals from said inertial sensor (106) representative of user movement, convert said signals into control signals for manipulating said computer game, said control signals being of the same format as control signals generated in response to user manipulation of a respective control input device, and cause said control signals to be transmitted to said control box (104).

An Internet of Thing (IoT) sport device includes a body with a processor, a camera and a wireless transceiver coupled to the processor.

Method for providing image of HMD user to a non-HMD user includes, receiving a first image of a user including the user's facial features captured by an external camera when the user is not wearing a head mounted display (HMD). A second image capturing a portion of the facial features of the user when the user is wearing the HMD is received. An image overlay data is generated by mapping contours of facial features captured in the second image with contours of corresponding facial features captured in the first image. The image overlay data is forwarded to the HMD for rendering on a second display screen that is mounted on a front face of the HMD.

A handheld terminal 200 performs wireless communication with a game apparatus 103 and determines whether the wireless communication with the game apparatus 103 is possible. The game apparatus 103 performs wireless communication with the handheld terminal 200 and determines whether the wireless communication with the handheld terminal 200 is possible. The game apparatus 103 performs a return home determination process, based on the determination about whether the wireless communication is possible, when the wireless communication with the handheld terminal 200 has become, after having become disabled, enabled again.

A handheld terminal 200 performs wireless communication with a game apparatus 103 and determines whether the wireless communication with the game apparatus 103 is possible. The game apparatus 103 performs wireless communication with the handheld terminal 200 and determines whether the wireless communication with the handheld terminal 200 is possible. The game apparatus 103 performs a return home determination process, based on the determination about whether the wireless communication is possible, when the wireless communication with the handheld terminal 200 has become, after having become disabled, enabled again.

A method to identify positions of fingers of a hand is described. The method includes capturing images of a first hand using a plurality of cameras that are part of a wearable device. The wearable device is attached to a wrist of a second hand and the plurality of cameras of the wearable device is disposed around the wearable device. The method includes repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in a head mounted display (HMD). The method includes sending the stream of captured image data to a computing device that is interfaced with the HMD. The computing device is configured to process the captured image data to identify changes in positions of the fingers of the first hand.