The A method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device is disclosed . The method includes determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment. The method further includes sending the CPI to a remote server. The method further includes receiving a rendered view of the volumetric video object that has been rendered in accordance with the CPI from the remote server. The method also includes augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

Systems and methods and computer program products for processing three-dimensional (3D) graphics are provided. A method includes receiving 3D geometry data for a shape to be rendered to a display that comprises an array of hogels, the shape defined in a model space. The method can further include reducing downstream processing of the 3D geometry data to render the shape to the display, comprising identifying a subset of hogels in a hogel plane that have hogel bowtie frustums that intersect the shape.

Systems and methods and computer program products for processing three-dimensional (3D) graphics are provided. A method includes receiving 3D geometry data for a shape to be rendered to a display that comprises an array of hogels, the shape defined in a model space. The method can further include reducing downstream processing of the 3D geometry data to render the shape to the display, comprising identifying a subset of hogels in a hogel plane that have hogel bowtie frustums that intersect the shape.

A three-dimensional data encoding method includes: extracting, from first three-dimensional data, second three-dimensional data having an amount of a feature greater than or equal to a threshold; and encoding the second three-dimensional data to generate first encoded three-dimensional data. For example, the three-dimensional data encoding method may further include encoding the first three-dimensional data to generate the second encoded three-dimensional data.

A three-dimensional data encoding method includes: extracting, from first three-dimensional data, second three-dimensional data having an amount of a feature greater than or equal to a threshold; and encoding the second three-dimensional data to generate first encoded three-dimensional data. For example, the three-dimensional data encoding method may further include encoding the first three-dimensional data to generate the second encoded three-dimensional data.

Embodiments of the present disclosure provide a software program that displays both a volume as images and segmentation results as surface models in 3D. Multiple 2D slices are extracted from the 3D volume. The 2D slices may be interactively rotated by the user to best follow an oblique structure. The 2D slices can “cut” the surface models from the segmentation so that only half of the models are displayed. The border curves resulting from the cuts are displayed in the 2D slices. The user may click a point on the surface model to designate a landmark point. The corresponding location of the point is highlighted in the 2D slices. A 2D slice can be reoriented such that the line lies in the slice. The user can then further evaluate or refine the landmark points based on both surface and image information.

Embodiments of the present disclosure provide a software program that displays both a volume as images and segmentation results as surface models in 3D. Multiple 2D slices are extracted from the 3D volume. The 2D slices may be interactively rotated by the user to best follow an oblique structure. The 2D slices can “cut” the surface models from the segmentation so that only half of the models are displayed. The border curves resulting from the cuts are displayed in the 2D slices. The user may click a point on the surface model to designate a landmark point. The corresponding location of the point is highlighted in the 2D slices. A 2D slice can be reoriented such that the line lies in the slice. The user can then further evaluate or refine the landmark points based on both surface and image information.

Provided is an image processing device including a matching unit that performs matching processing between a predetermined pattern on a surface of a 3D model of a biological tissue including an operating site generated on the basis of a preoperative diagnosis image and a predetermined pattern on a surface of the biological tissue included in a captured image during surgery, a shift amount estimation unit that estimates an amount of deformation from a preoperative state of the biological tissue on the basis of a result of the matching processing and information regarding a three-dimensional position of a photographing region which is a region photographed during surgery on the surface of the biological tissue, and a 3D model update unit that updates the 3D model generated before surgery on the basis of the estimated amount of deformation of the biological tissue.