Disclosed herein are systems and method for determining environment dimensions based on landmark detection, the method including: training, with a dataset including a plurality of images featuring an environment and labelled landmarks in the environment, a neural network to identify the labelled landmarks in an arbitrary image of the environment; receiving an input image depicting the environment; generating an input tensor based on the received input image; inputting the input tensor into the neural network, wherein the neural network is configured to generate an output tensor including a position of each identified landmark and a visibility score associated with each position; calculating a homography matrix between each position in the output tensor along a camera plane and a corresponding position in an environment plane, based on a pre-built model of the environment; and outputting an image that visually connects each landmark along the environment plane based on the homography matrix.

Augmented reality systems provide graphics over views from a mobile device for both in-venue and remote viewing of a sporting or other event. A server system can provide a transformation between the coordinate system of a mobile device (mobile phone, tablet computer, head mounted display) and a real world coordinate system. Requested graphics for the event are displayed over a view of an event. In a tabletop presentation, video of the event can be displayed with augmented reality graphics overlays at a remote location.

Augmented reality systems provide graphics over views from a mobile device for both in-venue and remote viewing of a sporting or other event. A server system can provide a transformation between the coordinate system of a mobile device (smart phone, tablet computer, head mounted display) and a real world coordinate system. Requested graphics for the event are displayed over a view of an event.

An information processing apparatus includes a detection unit that detects a three-dimensional position and a posture of an object in an instruction three-dimensional region having an enlarged or reduced relationship with an observation three-dimensional region in which a virtual viewpoint and a virtual visual line are defined, a derivation unit that derives the viewpoint and the visual line corresponding to detection results of the detection unit depending on positional relationship information indicating a relative positional relationship between the observation three-dimensional region and the instruction three-dimensional region, and an acquisition unit that acquires a virtual viewpoint image showing a subject in a case in which the subject is observed with the viewpoint and the visual line derived by the derivation unit, the virtual viewpoint image being based on a plurality of images obtained by imaging an imaging region included in the observation three-dimensional region by a plurality of imaging apparatuses.

Methods and systems are described for registering a sports field to a video. Video of a live event may feature participants at a venue. A template of the venue, including virtual markings that represent real markings on the venue, may be obtained. A homographic transformation between an image plane and a ground plane may be determined by matching virtual markings to corresponding real markings captured in at least one frame of the video. The determined homographic transformation may be used in the automated analysis of sports statistics and in improving inserted annotations and visualizations.

A system described herein may provide a technique for the real-time determination of events, objects, focal points, or the like to be captured by one or more cameras in a multi-camera environment. Such determination may be based on “crowdsourced” data from multiple User Equipment (“UEs”). The crowdsourced data may include positioning and/or pose information associated with UEs. The positioning information for a given UE may include location information, and the pose information may include an azimuth angle, magnetic declination, or other suitable information indicating where a particular physical facet of the UE is facing. For example, the pose information may be used to indicate or infer where a camera of the UE is pointed. One or more actuatable cameras may be displaced, rotated, etc. to capture video at one or more identified crowdsourced focal points.

A system and method for model-based camera tracking and image occlusion removal for a camera viewing a sports field (or other scene) includes receiving a synthesized data set comprising at least one empty field image of the field, the empty field image with at least one occlusion graphic, and camera parameters corresponding to the empty field image, training a neural network model to estimate the empty field image and the corresponding camera parameters by providing the model with an input training image comprising the empty field image with occlusion graphic, and providing the model with model output targets comprising the empty field image and the corresponding camera parameters as targets for the model, receiving by the neural network model, alive input image comprising a view of the field with live occlusions, and providing by the neural network model, using trained model parameters, estimated live camera parameters or an estimated empty field image associated with the live input image.

An exemplary method includes an augmented reality system acquiring an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area; identifying at least part of the performance area of the event within the image of the event; determining a three-dimensional (3D) pose of the performance area of the event within the image of the event; and providing, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.

An exemplary method includes an augmented reality system acquiring an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area; identifying at least part of the performance area of the event within the image of the event; determining a three-dimensional (3D) pose of the performance area of the event within the image of the event; and providing, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.

A method of capturing an image of a scene. A current location of a plurality of objects in a frame of a video capturing the scene having one or more events of interest, is determined. For at least one of the events of interest, a time and a location for each of the plurality of objects associated with the event of interest is predicted based on the current location of the plurality of objects. A frame subset score is determined for each of a plurality of frame subsets in the frame, each of the plurality of frame subsets including one or more of the plurality of objects based on the predicted time and the predicted location for the event of interest. One of the determined plurality of frame subsets is selected based on the determined frame subset score. An image of the event of interest is captured using a camera, based on a camera orientation setting for the selected frame subset, where the captured image comprises the selected frame subset.