A special test target may enable standardized testing of performance of image based depth measuring systems. In addition, the error in measured depth with respect to the ground truth may be used as a metric of system performance. This test target may aid in identifying the limitations of the disparity estimation algorithms.

Embodiments allow live action images from an image capture device to be composited with computer generated images in real-time or near real-time. The two types of images (live action and computer generated) are composited accurately by using a depth map. In an embodiment, the depth map includes a “depth value” for each pixel in the live action image. In an embodiment, steps of one or more of feature extraction, matching, filtering or refinement can be implemented, at least in part, with an artificial intelligence (AI) computing approach using a deep neural network with training. A combination of computer-generated (“synthetic”) and live-action (“recorded”) training data is created and used to train the network so that it can improve the accuracy or usefulness of a depth map so that compositing can be improved.

Apparatus comprises a camera configured to capture images of a user in a scene; a depth detector configured to capture depth representations of the scene, the depth detector comprising an emitter configured to emit a non-visible signal; a mirror arranged to reflect at least some of the non-visible signal emitted by the emitter to one or more features within the scene that would otherwise be occluded by the user and to reflect light from the one or more features to the camera; a pose detector configured to detect a position and orientation of the mirror relative to at least one of the camera and depth detector; and a scene generator configured to generate a three-dimensional representation of the scene in dependence on the images captured by the camera and the depth representations captured by the depth detector and the pose of the mirror detected by the pose detector.

A method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, comprises: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine a camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

A method and apparatus for outputting a three-dimensional (3D) image are provided. To output a 3D image, a stereo image is generated based on viewpoints of a user and rendered into a 3D image. Since the stereo image is generated based on the viewpoints of the user, the user views a different side of an object appearing in the 3D image depending on a viewpoint of the user.

A stereo image matching apparatus includes a processor which includes: a bit distributor distributing values of each pixel of stereo images into sequential N bits and outputting a plurality of stereo images including the sequential N bits; a plurality of cost calculators each receiving the plurality of stereo images and calculating matching cost values for each pixel of each of the stereo images; a confidence calculator calculating a matching confidence by using cost characteristics lit of the respective matching cost values calculated by the plurality of cost calculators; and a depth determiner determining that a depth value of which the matching confidence is high and the matching cost values are relatively low is a final depth value.

The present invention includes computer vision based driver assistance devices, systems, methods and associated computer executable code (hereinafter collectively referred to as: “ADAS”). According to some embodiments, an ADAS may include one or more fixed image/video sensors and one or more adjustable or otherwise movable image/video sensors, characterized by different dimensions of fields of view. According to some embodiments of the present invention, an ADAS may include improved image processing. According to some embodiments, an ADAS may also include one or more sensors adapted to monitor/sense an interior of the vehicle and/or the persons within. An ADAS may include one or more sensors adapted to detect parameters relating to the driver of the vehicle and processing circuitry adapted to assess mental conditions/alertness of the driver and directions of driver gaze. These may be used to modify ADAS operation/thresholds.

A system for executing a three-dimensional (3D) intraoperative scan of a patient is disclosed. A 3D scanner controller projects the object points included onto a first image plane and the object points onto a second image plane. The 3D scanner controller determines first epipolar lines associated with the first image plane and second epipolar lines associated with the second image plane based on an epipolar plane that triangulates the object points included in the first 2D intraoperative image to the object points included in the second 2D intraoperative image. Each epipolar lines provides a depth of each object as projected onto the first image plane and the second image plane. The 3D scanner controller converts the first 2D intraoperative image and the second 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point provided by each corresponding epipolar line.

Embodiments of the invention provide a camera array imaging architecture that computes depth maps for objects within a scene captured by the cameras, and use a near-field sub-array of cameras to compute depth to near-field objects and a far-field sub-array of cameras to compute depth to far-field objects. In particular, a baseline distance between cameras in the near-field subarray is less than a baseline distance between cameras in the far-field sub-array in order to increase the accuracy of the depth map. Some embodiments provide an illumination near-IR light source for use in computing depth maps.

A device for detecting the surroundings of a vehicle and a method for detecting the surroundings, and a vehicle designed to carry out said method comprise a camera module, a camera control apparatus, an analysis unit and an illumination device. The illumination device is formed by a matrix headlight of the vehicle and is designed such that it can project a light pattern into the surroundings. The projected light pattern is imaged in the detection region of the camera module and the 3D position of measurement points formed by the light pattern in the surroundings is determined by the analysis unit. However, the illumination device projects the light pattern only into regions of the surroundings in which the analysis unit has ascertained, based on image data, a value that is critical for 3D position determination.