An image processing device includes; an image sensor including pixels arranged in a non-Bayer pattern and configured to generate non-Bayer pattern image data in response to electrical signals generated by the pixels, and an image signal processor configured to determine an indexing value indicating a color of a target pixel among the pixels and directly process the non-Bayer pattern image data in response to the indexing value.

Embodiments relate to a multi-mode demosaicing circuit able to receive and demosaic image data in a different raw image formats, such as Bayer raw image format and Quad Bayer raw image format. The multi-mode demosaicing circuit demosaics Quad Bayer image data by interpolating a green channel of the image data along each of a plurality of directions, generating a gradient of the image data along each of the plurality of directions, modifying the interpolated green channels based on respective gradients to generate full-resolution green channel image data, which is combined with red and blue image data to generate the demosaiced image data. Interpolation is performed for non-green pixels based on neighboring green pixels along a specified direction, modified by a residual value based upon values of one or more nearby same-color pixels and a correlation between values of the same-color pixels and neighboring green pixels.

Provided is an apparatus for acquiring images including an image sensor including a sensor substrate including a plurality of photo-sensing cells sensing light, and a color separation lens array provided above the sensor substrate, the color separation lens array including a fine structure in each of a plurality of regions respectively facing the plurality of photo-sensing cells and separating incident light based on color, the fine structure forming a phase distribution to condense light having different wavelengths on adjacent photo-sensing cells, a signal processor configured to perform, based on a point spread function corresponding to each color pixel by the color separation lens array, deconvolution on sensing signals of the plurality of photo-sensing cells to process an image signal for each color obtained by the image sensor, and an image processor configured to form a color image from the image signal for each color processed by the signal processor.

An image postprocessor enhances the quality and detail of a rendered image by applying a photosite demosaicing technique directly to the pixels stored in the viewport of the rendered image. The image postprocessor detects a first subset of the viewport pixels having color values that deviate by more than a threshold amount from color values of a neighboring second subset of the viewport pixels. The image postprocessor maps the viewport pixels to a set of emulated photosites based on a position of each pixel in the viewport, and demosaics and/or interpolates the color values associated with a subset of the emulated photosites in order to generate new pixels for the gap. The image postprocessor replaces the first subset of pixels in the viewport with the new pixels, and presents a second visualization based on the modified pixels of the viewport.

Embodiments relate to a front-end scaler circuit configured to receive and process demosaiced image data in different modes depending on if the demosaiced image data was demosaiced from Bayer or Quad Bayer raw image data. The front-end scaler circuit shares memory with a demosaicing circuit, and is configured to perform different operations that use different amounts of the shared memory based on the original image format of the demosaiced image data being processed, to compensate for additional memory utilized by the demosaicing circuit when demosaicing certain types of image data. For example, when processing image data demosaiced from Quad Bayer image data, the front-end scaler circuit discards a portion of the chrominance component data for the received image data before performing chromatic suppression, compared to when processing image data demosaiced from Bayer image data.

In some implementations, a method includes: determining a complexity value for first image data associated with of a physical environment that corresponds to a first time period; determining an estimated composite setup time based on the complexity value for the first image data and virtual content for compositing with the first image data; in accordance with a determination that the estimated composite setup time exceeds the threshold time: forgoing rendering the virtual content from the perspective that corresponds to the camera pose of the device relative to the physical environment during the first time period; and compositing a previous render of the virtual content for a previous time period with the first image data to generate the graphical environment for the first time period.

An image is received from a camera built into a cabin of a vehicle. The image is demosaiced and its noise is reduced. A segmentation algorithm is applied to the image. A global illumination for the image is solved. Based on the segmentation of the image and the global illumination, a bidirectional reflectance distribution function (BRDF) for color and/or reflectance information of material in the cabin area of the vehicle is solved for. A white balance matrix and a color correction matrix for the image are computed based on the BRDF. The white balance matrix and the color correction matrix are applied to the image, which is then displayed or stored for addition image processing.

An information processing apparatus includes an interpolation target color difference pixel generator that generates, on the basis of a relationship between a luminance pixel at an interpolation target pixel position that is a pixel position at which a first color difference pixel does not exist and the luminance pixel at at least one neighboring pixel position of a plurality of pixel positions near the interpolation target pixel position in image data, an interpolation target color difference pixel corresponding to the first color difference pixel at the interpolation target pixel position, in which the image data is generated on the basis of three primary color pixels that can include a value greater than a predetermined white clip value, the image data having a number of the luminance pixels larger than a number of the first color difference pixels and a number of second color difference pixels.

Techniques for generating a high resolution full color output image from lower resolution sparse color input images are disclosed. A camera generates images. The camera's sensor has a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and IMU data are fed into a motion model, which performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, a blue-only image, and a monochrome image. The color images are up-sampled to match the resolution of the monochrome image. A high resolution output color image is generated by combining the up-sampled images and the monochrome image.

Techniques for using motion data to generate a high resolution output color image from multiple images having sparse color information are disclosed. A camera generates multiple images. The camera's sensor is configured to have a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and the IMU data are fed as input into a motion model. The motion model performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, and a blue-only image. A high resolution output color image is generated by combining the red-only image, the green-only image, and the blue-only image.