IMAGE OPTIMIZATION IN MOBILE CAPTURE AND EDITING APPLICATIONS

Abstract

HDR color patches are sampled throughout an HDR color space parameterized by a parameter. Reference SDR color patches, input HDR color patches and reference HDR color patches are generated from the sampled HDR color patches. An optimization algorithm is executed to generate an optimized forward reshaping mapping and an optimized backward reshaping mapping. The optimized forward reshaping mapping is used to forward reshape input HDR images into forward reshaped SDR images, whereas the optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR images into backward reshaped HDR images.

Claims

1. A method comprising: extracting a set of standard dynamic range (SDR) image feature points from a training SDR image and extracting a set of high dynamic range (HDR) image feature points from a training HDR image; matching a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points; using the subset of one or more SDR image feature points and the subset of one or more HDR image feature points to generate a geometric transform to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image; determining a set of pairs of SDR and HDR color patches, from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR and HDR images have been spatially aligned by the geometric transform; generating an optimized SDR-to-HDR mapping, based at least in part on the set of pairs of SDR and HDR color patches derived from the training SDR image and the training HDR image; applying the optimized SDR-to-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

2. The method of claim 1, wherein the training SDR image and the training HDR image are captured by a capturing device operating in SDR and HDR capture modes, respectively, from a three-dimensional (3D) visual scene.

3. The method of claim 1, wherein the training SDR image and the training HDR image form a pair of training SDR and HDR images in a plurality of pairs of training SDR and HDR images; wherein the optimized SDR-to-HDR mapping is generated based at least in part on a plurality of sets of pairs of SDR and HDR color patches derived from the plurality of pairs of training SDR and HDR images.

4. The method of claim 1, wherein each SDR image feature point in the subset of one or more SDR image feature points is matched with a respective HDR image feature point in the subset of one or more HDR image feature points; wherein the SDR image feature point and the HDR image feature point are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

5. The method of claim 1, wherein the SDR training image and the HDR training image are obtained by performing respective camera distortion correction operations on each distorted training image in a pair of a distorted training standard dynamic range (SDR) image and a distorted training high dynamic range (HDR) image to generate a respective training image in a pair of a training SDR image and a training HDR image; wherein the set of SDR image feature points and the set of HDR image feature points correspond to corner pattern marks; wherein using the subset of one or more SDR image feature points and the subset of the one or more HDR image feature points comprise generating a respective projective transform, in a pair of an SDR image projective transform and an HDR image projective transform, using the corner pattern marks detected from each training image in the pair of the training SDR image and the training HDR image.

6. The method of claim 5, wherein the training SDR image and the training HDR image are captured by a first capturing device operating in an SDR capture mode and a second capturing device operating in an HDR capture mode, respectively, from a common color chart image.

7. The method of claim 5, wherein the common color chart image is rendered on and captured by the first capturing device and the second capturing device from a screen of a common reference image display.

8. The method of claim 5, wherein the respective camera distortion correction operations are based at least in part on camera-specific distortion coefficients generated from a camera calibration process performed with a camera used to acquire the training image.

9. The method of claim 5, wherein the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3DMT); wherein the optimized SDR-to-HDR mapping is generated based at least in part on the 3DMT.

10. The method of claim 5, wherein the optimized SDR-to-HDR mapping represents one of: tensor-product B-Spline (TPB) based mapping or a non-TPB-based mapping.

11. A method comprising: building sampled high dynamic range (HDR) color space points distributed throughout an HDR color space used to represent reconstructed HDR images; converting the sampled HDR color space points into standard dynamic range (SDR) color space points in a first SDR color space in which SDR images to be edited by an editing device are represented; determining a bounding SDR color space rectangle based on extreme SDR codeword values of the SDR color space points in the first SDR color space and determining an irregular three-dimensional (3D) shape from a distribution of the SDR color space points; building sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space; using the sampled SDR color space points and the irregular shape to generate a boundary clipping 3D lookup table (3D-LUT) including lookup keys and corresponding lookup values, wherein the boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys; wherein, when the lookup key is within the irregular shape, the lookup key equals the lookup value; wherein, when the lookup key is outside the irregular shape, the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns a nearest neighbor inside the irregular shape to the lookup key as lookup value; performing clipping operations, based at least in part on the boundary clipping 3D-LUT, on an edited SDR image in the first SDR color space to generate a boundary clipped edited SDR image in the first SDR color space.

12. The method of claim 11, wherein the clipping operations includes first using the bounding SDR color space rectangle to perform regular clipping on the edited SDR image to generate a regularly clipped edited SDR image and subsequently using the 3D-LUT to perform irregular clipping on the regularly clipped edited SDR image to generate the boundary clipped edited SDR image.

13. An apparatus comprising a processor and configured to perform the method recited in claim 1.

14. A non-transitory computer-readable storage medium having stored thereon computer-executable instruction for executing a method with one or more processors in accordance with the method recited in claim 1.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] An embodiment of the present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0011] FIG. 1A through FIG. 1E illustrate example process flows for applying optimized reshaping operations;

[0012] FIG. 2A illustrates an example color distribution of an irregular shape in relation to HDR and SDR color spaces such as R.2020, P3 and R.709 color spaces; FIG. 2B through FIG. 2E and FIG. 2H through FIG. 2K illustrate example parameterized color spaces in the R.2020 color space; FIG. 2F and FIG. 2G illustrate example percentile Cb and Cr values in forward reshaped SDR color spaces corresponding to different parameterized HDR color spaces; FIG. 2L illustrates an example color chart image; FIG. 2M and FIG. 2N illustrate example checkerboard images; FIG. 2O illustrates an example original color chart image displayed on the reference image display, an example captured image from the color chart image and a rectified captured image generated from the captured image; FIG. 2P illustrates an example distribution of RGB values; FIG. 2Q through FIG. 2T illustrate example alpha shapes;

[0013] FIG. 3A and FIG. 3C illustrate example process flows for finding an optimized color space to represent HDR colors; FIG. 3B illustrates an example process flow for determining optimized values for programmable parameters in an ISP pipeline; FIG. 3D illustrates an example process flow for generating optimized reshaping operational parameters; FIG. 3E illustrates an example process flow to generate a plurality of different color charts; FIG. 3F illustrates an example process flow for matching SDR and HDR colors between an image pair of captured SDR and HDR images; FIG. 3G and FIG. 3H illustrate example encoder-side and decoder-side image/video editing operations; FIG. 3I through FIG. 3M illustrate example solutions for clipping out-of-range codewords in image/video editing applications;

[0014] FIG. 4A through FIG. 4E illustrate example process flows; and

[0015] FIG. 5 illustrates a simplified block diagram of an example hardware platform on which a computer or a computing device as described herein may be implemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

[0016] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present disclosure.

SUMMARY

[0017] Image optimizations such as those relating to tensor-product B-Spline (TPB) based image reshaping solutions are described herein for video capture applications including but not limited to mobile (video) capture applications. The TPB based solutions can provide or generate relatively accurate reconstructed images using robust predictors implemented with video codecs such as backward compatible video codecs. These predictors can operate with static mapping to gracefully handle or recover from transmission errors such as frame dropping, minimize power consumption such as battery power consumption, reduce data overheads and/or processing/transmission latencies, etc. In some operational scenarios, some or all of the TPB based solutions as described herein may be implemented with mobile applications that are to comply with relatively harsh battery or power constraint.

[0018] The TPB based image reshaping solutions can be adapted to operating in different use cases. Some of the use cases allow freedom to design and/or apply HDR-to-SDR mappings to generate SDR images to be encoded in (base layers of) video signals as described herein. Some of the use cases rely on mobile image signal processors (ISPs) with relatively limited programmable registers to generate SDR images to be encoded in (base layers of) video signals. Different architectures may be used to implement the TPB based solutions as described herein. Additionally, optionally or alternatively, these architectures or the TPB based solutions can be used to provide or support backward compatibility.

[0019] Given reference SDR images represented in an SDR color space or a color space in an SDR domain), the TPB-based solutions can use a TPB optimization process to achieve or determine a maximum or widest HDR color space or a color space in an HDR domain to represent corresponding reconstructed HDR images predicted from the SDR images. The wider the HDR color space, the more color deviations introduced in the SDR color space. The TPB optimization process can be implemented to achieve or determine an optimized balance point or tradeoff between achieving the maximum or widest HDR color space on one hand and introducing SDR color deviations on the other. Additionally, optionally or alternatively, the TPB optimization process as described herein can be implemented to incorporate neutral color processing or perform constraint optimization to help preserve gray levels represented in the SDR images in the reconstructed HDR images predicted from the SDR images.

[0020] As more and more videos are being captured by various camera-equipped computing or mobile devices in practice, video editing are also becoming more and more popular. Users operating these devices may be allowed to adjust the look of captured videos manually or simply apply default theme templates to the captured videos. Depending on available computation resources and/or preferences of video editing tool designers/providers, video editing may be done in either a source domain such as a source HDR domain in which source images are represented or a non-source domain such as a generated SDR domain to which pre-edited images may be converted from the source images in the source domain.

[0021] In operational scenarios in which SDR images are to be encoded in a video signal, video editing operations in the HDR domain or the source domain will not adversely impact downstream devices of the video signal to reconstruct HDR images from the SDR images decoded from the video signal at the decoder side. This is so because these video editing operations do not interfere HDR-to-SDR mappings to generate the SDR images from HDR images or source images in the HDR domain at the encoder side, and also do not interfere SDR-to-HDR mappings to map back to or reconstruct an approximation of, the HDR images at the decoder side.

[0022] However, in these operational scenarios, video editing operations in the SDR domain will likely adversely impact revertability between the SDR domain and the HDR domain. For example, the edited SDR images may not be able to be mapped back to the original or source HDR images. Furthermore, a color space to represent the SDR images may be limited with pre-defined ranges such as SMPTE ranges that pixel values or codewords cannot exceed. Color space conversion such as YUV-to-RGB conversion used for video editing operations may cause pixel values or codewords to be clipped and hence damaging or harming the reversibility. Clipping techniques as described herein can be used to reduce or prevent adverse editing impacts on video editing operations including but not limited to those performed with mobile devices. A two-level TPB boundary clipping may be implemented in the RGB domain to support or preserve maximum edited colors in the SDR domain that can be used to propagate or map back to reconstructed images in the HDR domain.

[0023] Example embodiments described herein relate to image generation. Sampled HDR color space points distributed throughout an HDR color space are built. The HDR color space is parameterized by a color primary scaling parameter with a candidate value selected from among a plurality of candidate values. The color primary scaling parameter is used to compute color space coordinates of at least one of multiple color primaries delineating the HDR color space. Reference SDR color space points represented in a reference SDR color space, input HDR color space points represented in an input HDR color space, and reference HDR color space points represented in a reference HDR color space points, are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is executed to generate a chain of an optimized forward reshaping mapping and an optimized backward reshaping mapping. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points and the reference HDR color space points as input. The optimized forward reshaping mapping is used to forward reshape input HDR images in the input HDR color space into forward reshaped SDR images in a forward reshaped SDR color space, whereas the optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR images in the forward reshaped SDR color space into backward reshaped HDR images.

[0024] Example embodiments described herein relate to image generation. Sampled HDR color space points distributed throughout an HDR color space are built. The HDR color space is parameterized by a color primary scaling parameter with a candidate value selected from among a plurality of candidate values. The color primary scaling parameter is used to compute color space coordinates of at least one of multiple color primaries delineating the HDR color space. Input SDR color space points represented in an input SDR color space and reference HDR color space points represented in a reference HDR color space points are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is executed to generate an optimized backward reshaping mapping. The reshaping operation optimization algorithm receives the input SDR color space points and the reference HDR color space points as input. The backward reshaping mapping is used to backward reshape SDR images in the input SDR color space into backward reshaped HDR images.

[0025] Example embodiments described herein relate to image generation. A set of SDR image feature points is extracted from a training SDR image, whereas a set of HDR image feature points is extracted from a training HDR image. A subset of one or more SDR image feature points in the set of SDR image feature points is matched with a subset of one or more HDR image feature points in the set of HDR image feature points. The subset of one or more SDR image feature points and the subset of one or more HDR image feature points are used to generate a geometric transform to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image. A set of pairs of SDR and HDR color patches is determined from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR and HDR images have been spatially aligned by the geometric transform. An optimized SDR-to-HDR mapping is generated based at least in part on the set of pairs of SDR and HDR color patches derived from the training SDR image and the training HDR image. The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0026] Example embodiments described herein relate to image generation. Respective camera distortion correction operations are performed on each training image in a pair of a training standard dynamic range (SDR) image and a training high dynamic range (HDR) image to generate a respective undistorted image in a pair of an undistorted training SDR image and an undistorted training HDR image. A respective projective transform, in a pair of an SDR image projective transform and an HDR image projective transform, is generated using corner pattern marks detected from each undistorted image in the pair of the undistorted training SDR image and the undistorted training HDR image. Each projective transform in the pair of the SDR image projective transform and the HDR image projective transform is applied to a respective undistorted image in the pair of the undistorted training SDR image and the undistorted training HDR image to generate a respective rectified image in a pair of a rectified training SDR image and a rectified training HDR image. A set of SDR color patches is extracted from the rectified training SDR image, whereas a set of HDR color patches is extracted from the rectified training HDR image. An optimized SDR-to-HDR mapping is generated based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image. The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

[0027] Example embodiments described herein relate to clipping operations on edited images. Sampled HDR color space points distributed throughout an HDR color space used to represent reconstructed HDR images are built. The sampled HDR color space points are converted into SDR color space points in a first SDR color space in which SDR images to be edited by an editing device are represented. A bounding SDR color space rectangle is determined based on extreme SDR codeword values of the SDR color space points in the first SDR color space. An irregular three-dimensional (3D) shape is determined from a distribution of the SDR color space points. Sampled SDR color space points distributed throughout the bounding SDR color space rectangle in the first SDR color space are built. The sampled SDR color space points and the irregular shape are used to generate a boundary clipping 3D lookup table (3D-LUT). The boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys. Clipping operations are performed, based at least in part on the boundary clipping 3D-LUT, on an edited SDR image in the first SDR color space to generate a boundary clipped edited SDR image in the first SDR color space.

Reshaping Optimization in Image/Video Capture Applications

[0028] Reshaping optimization processes such as TPB and/or non-TPB optimization processes can be implemented or incorporated in video capture applications running on computing devices such as mobile devices in a variety of operational scenarios. The video capture applications with the reshaping optimization processes can be used to generate or output video signals such as base layer or SDR images encoded therein. The reshaping optimization processes can implement different solutions in different operational scenarios to generate or optimize reshaping operational parameters such as TPB and/or non-TPB coefficients to be used with the base layer or SDR images to generate, construct or reconstruct non-base-layer or HDR images with optimized picture quality.

[0029] For the purpose of illustration, the reshaping optimization processes, or the solutions implemented therein, may be classified into different types based on specific SDR generation processes or sub-processes adopted by the video capturing applications as well as based on specific reshaping path(s)forward (reshaping) path and/or backward (reshaping) pathsubjected to reshaping optimization.

[0030] FIG. 1A illustrates a first example reshaping optimization process implementing a white-box joint forward and backward TPB optimization design/solution (referred to as WFB) with a (single) video capture device. The white-box joint forward and backward TPB optimization design/solution may be implemented for operational scenarios in which a SDR generation process or sub-process converts reference HDR images into reference SDR images with white-box conversion operations and in which both forward and backward (reshaping) paths can be subjected to TPB optimization.

[0031] As used herein, white-box conversion refers to HDR-to-SDR mapping or conversion with well-defined conversion or mapping functions/equations such as those (e.g., publicly, etc.) specified or documented in standard-based or proprietary video coding specifications. In comparison, black-box conversion refers to HDR-to-SDR mapping or conversion with conversion or mapping operations not based on the well-defined conversion or mapping functions/equations. For example, black-box conversion may be implemented as internal image signal processes performed by image signal processors with no or little reliance on any well-defined conversion or mapping functions or equations (e.g., publicly, etc.) specified or documented in standard-based or proprietary video coding specifications.

[0032] By way of example but not limitation, in FIG. 1A, an input video signal including the HDR images may be represented in an input color space such as a full R.2020 color space. A generated SDR signal including the reference SDR images may be represented in a reference SDR color space. The reference SDR images can be images generated from the HDR images with publicly known HDR-to-SDR mapping processes, functions and/or equations.

[0033] As shown in FIG. 1A, in the forward reshaping path, the reference HDR images may be forward reshaped into (forward) reshaped SDR images approximating the reference SDR images, based at least in part on optimized forward reshaping operational parameters (denoted as forward TPB optimization) generated from a joint forward-and-backward TPB optimization solution. The reshaped SDR images may be represented in a reshaped SDR color space and generated by an upstream device to be included/encoded in a video signalor a base layer (BL) thereofoutputted by the upstream device.

[0034] A downstream recipient device or a video decoder of the video signal can decode the reshaped SDR images from the video signal. The decoded reshaped SDR images at the decoder side may be the same as the reshaped SDR images at the encoder side, subject to errors introduced in compression/decompression, coding operations and/or data transmissions.

[0035] In the backward reshaping path as implemented by the downstream device, the reshaped SDR images may be backward reshaped into (backward) reshaped HDR images based at least in part on optimized backward reshaping operational parameters (denoted as backward TPB optimization) generated from the same joint forward-and-backward TPB optimization solution. The reshaped HDR imagesgenerated with the optimized backward reshaping operational parameters in an output HDR color spacerepresent an approximation to or reconstructed version of the reference HDR images.

[0036] The joint forward-and-backward TPB optimization solution can generate the optimized forward and backward reshaping operational parameters to cover as wide as possible in the output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0037] One or both of the optimized forward and backward reshaping operational parameters such as optimized forward and backward TPB coefficients can be implemented or represented in three-dimensional look-up table(s) or 3D-LUT(s) to reduce processing times in reshaping operations.

[0038] Since the forward and backward reshaping operational parameters such as forward and backward TPB coefficients are jointly or concurrently designed or optimized in the joint forward and backward TPB optimization process, the supported reshaped SDR and HDR color spaces used to represent the reshaped SDR and HDR images can be jointly or concurrently designed or optimized in the same process.

[0039] In some operational scenarios, the optimized forward and backward reshaping operational parameters generated from the joint forward and backward TPB optimization process may be applied in a static single-layer backward compatible (SLBC) framework. Under this static framework, there is no need to (e.g., dynamically, etc.) obtain optimize image-specific or image-dependent optimized forward and backward operational parameters such as image-specific or image-dependent (or content-dependent) forward and backward TPB coefficients, for example on the fly while images are being processed.

[0040] Rather, under the static SLBC framework, the same or static optimized forward and backward operational parameters such as the same optimized forward and backward TPB coefficients can be obtained or generated oncefor example, offline or before performing reshaping operations on any of the (input) reference HDR images or the reshaped SDR imagesfor forward reshaping all the (input) reference HDR images and backward reshaping the reshaped SDR images. In an example, a single set of static optimized forward and backward operational parameters can be generated offline by a system as described herein and configured/deployed in or used by a capturing device as described herein to perform image reshaping or reconstruction operations. In another example, multiple sets of static optimized forward and backward operational parameters can be generated offline by a system as described herein and configured/deployed in or used by a capturing device as described herein to select a specific set of static optimized forward and backward operational parameters to perform image reshaping or reconstruction operations.

[0041] Thereafter, the optimized static forward TPB coefficients can be applied by the upstream device to forward reshape some or all of the (input) reference HDR images (e.g., a sequence of consecutive or sequential (input) reference HDR images, etc.) to generate the reshaped SDR images to be encoded in the (SLBC) video signal, whereas the optimized static backward TPB coefficients can be applied by the downstream recipient device of the video signal to some or all of the reshaped SDR images (e.g., a sequence of consecutive or sequential reshaped SDR images, etc.) decoded from the video signal to generate or reconstruct the reshaped HDR images.

[0042] FIG. 1B illustrates a second example reshaping optimization process implementing a white-box backward only TPB optimization design/solution (referred to as WB) with a (single) video capture device. The white-box backward only TPB optimization design/solution may be implemented for operational scenarios in which a SDR generation process or sub-process converts reference HDR images into reference SDR images with white-box conversion operations and in which only the backward (reshaping) path is subjected to TPB optimization.

[0043] By way of example but not limitation, in FIG. 1B, an input video signal including the HDR images may be represented in an input color space such as a P3 color space. A generated SDR signal including the reference SDR images may be represented in a reference SDR color space. The reference SDR images can be images generated from the HDR images with publicly known HDR-to-SDR mapping processes, functions and/or equations.

[0044] As shown in FIG. 1B, the reference HDR images may be processed by a programmable image signal processor or ISP (e.g., with given image signal processing functions/equations/operations, etc.) into ISP SDR images approximating the reference SDR images, based at least in part on optimized ISP operational parameters (denoted as ISP parameter optimization) generated from an ISP optimization solution. The ISP SDR images may be represented in an ISP SDR color space and generated by an upstream device to be included/encoded in a video signalor a base layer (BL) thereofoutputted by the upstream device.

[0045] A downstream recipient device or a video decoder of the video signal can decode the ISP SDR images from the video signal. The decoded ISP SDR images at the decoder side may be the same as the ISP SDR images at the encoder side, subject to errors introduced in compression/decompression, coding operations and/or data transmissions.

[0046] In the backward reshaping path as implemented by the downstream device, the ISP SDR images may be backward reshaped into (backward) reshaped HDR images based at least in part on optimized backward reshaping operational parameters (denoted as backward TPB optimization) generated from the backward only TPB optimization solution. The reshaped HDR imagesgenerated with the optimized backward reshaping operational parameters in an output HDR color spacerepresent an approximation to or reconstructed version of the reference HDR images.

[0047] The backward only TPB optimization solution can generate the optimized backward reshaping operational parameters to cover as wide as possible in the output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0048] The optimized backward reshaping operational parameters such as optimized backward TPB coefficients can be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce processing times in reshaping operations.

[0049] In some operational scenarios, the backward reshaping operational parameters generated from the backward only TPB optimization process may be applied in a static single-layer inverse display mapping (SLiDM) framework. Under this static framework, there is no need to (e.g., dynamically, etc.) obtain optimize image-specific or image-dependent optimized backward operational parameters such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0050] Rather, under the static SLiDM framework, the same or static optimized backward operational parameters such as the same optimized backward TPB coefficients can be obtained or generated oncefor example, offline or before performing reshaping operations on any of the reshaped SDR imagesfor backward reshaping the ISP SDR images. In an example, a single set of static optimized backward operational parameters can be generated offline by a system as described herein and configured/deployed in or used by a capturing device as described herein to perform image reshaping or reconstruction operations. In another example, multiple sets of static optimized backward operational parameters can be generated offline by a system as described herein and configured/deployed in or used by a capturing device as described herein to select a specific set of static optimized backward operational parameters to perform image reshaping or reconstruction operations.

[0051] Thereafter, the optimized static backward TPB coefficients can be applied by the downstream recipient device of the video signal to some or all of the ISP SDR images (e.g., a sequence of consecutive or sequential ISP SDR images, etc.) decoded from the video signal to generate or reconstruct the reshaped HDR images.

[0052] The ISP SDR images encoded in the video signal may or may not be identical to a desired SDR look such as represented by the reference images. The operational parameters or settings thereof in the programmable ISP can be optimized to approximate the reference images. Hence, in the WB operational scenarios of FIG. 1B, the SDR images for backward reshaping into the reshaped HDR images are given or fixed as the ISP SDR images. By way of comparison, in the WFB operational scenarios of FIG. 1A, the SDR images for backward reshaping into the reshaped HDR images are the forward reshaped SDR images, which can be optimized by applying the optimized forward reshaping operational parameters generated in the joint forward and backward TPB optimization process that generates both the optimized forward and backward reshaping parameters. In other words, TPB optimization is not used in the forward path to generate the ISP SDR images for the purpose of improving or enhancing both the desired look of the SDR images encoded in the video signal and the desired look of the reshaped HDR images. Relatively large deviations from the desired looks may occur in these operational scenarios.

[0053] FIG. 1C illustrates a third example reshaping optimization process implementing a black-box backward only TPB optimization design/solution (referred to as BB1) with a (single) video capture device. The black-box backward only TPB optimization design/solution may be implemented for operational scenarios in which SDR images are not generated from HDR images using white-box conversion operations and in which only the backward (reshaping) path is subjected to TPB optimization.

[0054] Reshaping operational parameters in these operational scenarios can be generated with training SDR images and training HDR images generated or acquired by the same video capture device such as the same mobile device. These training SDR images and training HDR images form a plurality of SDR and HDR image pair each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

[0055] A training SDR image and a training HDR image in the same SDR and HDR image pair may be acquired at different time instances/points (e.g., a few milliseconds apart, a few fractional seconds apart, a few seconds apart, etc.) using the same capturing device using the same ISP. The training SDR and HDR images may not be exactly spatially aligned or temporally aligned with each other, as it is difficult if not impossible to maintain the same shooting positions at the different time instances/points and process the training SDR and HDR images identically. For example, the training SDR image can be local tone mapped or local enhanced, whereas the training HDR image may be generated or obtained from multiple camera exposures. Accordingly, in the BB1 operational scenarios, relationships between pixel or codeword values in the training SDR image and corresponding pixel or codeword values in the training HDR image in the same image pair may be treated as, or assumed to be, a black box.

[0056] Training SDR and HDR images in each of the image pairs may be first spatially aligned. Spatially aligned training SDR and HDR images in the image pairs may be used to determine or find out matching color pairs. These matching color pairs can then be used to generate the optimized reshaping operational parameters such as the optimized TPB and/or non-TPB coefficients for reshaping or mapping (e.g., non-training, reference, etc.) SDR images back to backward reshaped or reconstructed HDR images approximating (e.g., non-training, reference, etc.) HDR images.

[0057] As shown in FIG. 1C, SDR images such as reference SDR images may be generated by an upstream device such as a capture device) to be included/encoded in a video signalor a base layer (BL) thereofoutputted by the upstream device. Example reference SDR images as described herein may include, but are not necessarily limited to only, SDR images generated from a programmable image signal processor of the upstream device or a capture device operating in conjunction with the upstream device.

[0058] A downstream recipient device or a video decoder of the video signal can decode the reference SDR images from the video signal. The decoded reference SDR images at the decoder side may be the same as the reference SDR images at the encoder side, subject to errors introduced in compression/decompression, coding operations and/or data transmissions.

[0059] In the backward reshaping path as implemented by the downstream device, the reference SDR images may be backward reshaped into (backward) reshaped HDR images based at least in part on optimized backward reshaping operational parameters (denoted as backward TPB optimization) generated from the black-box backward only TPB optimization design/solution. The reshaped HDR imagesgenerated with the optimized backward reshaping operational parameters in an output HDR color spacerepresent an approximation to or reconstructed version of reference HDR images that can or could be generated by the same capture device.

[0060] The black-box backward only TPB optimization design/solution can generate the optimized backward reshaping operational parameters to cover as wide as possible in the output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0061] The optimized backward reshaping operational parameters such as optimized backward TPB coefficients can be implemented or represented in a three-dimensional look-up table or 3D-LUT to reduce processing times in reshaping operations.

[0062] In some operational scenarios, the backward reshaping operational parameters generated from the black-box backward only TPB optimization design/solution may be applied in a SLiDM framework. Under this static framework, there is no need to (e.g., dynamically, etc.) obtain optimize image-specific or image-dependent optimized backward operational parameters such as image-specific or image-dependent (or content-dependent) backward TPB coefficients.

[0063] Rather, under the static SLiDM framework, the same or static optimized backward operational parameters such as the same optimized backward TPB coefficients can be obtained or generated oncefor example, offline or before performing reshaping operations on any of the reshaped SDR imagesfor backward reshaping the spatially aligned training SDR images generated by the upstream capture device to reshaped or reconstructed HDR images approximating the spatially aligned training HDR images generated by the same capture device.

[0064] Thereafter, the optimized static backward TPB coefficients can be applied by the downstream recipient device of the video signal to some or all of the (e.g., non-training, etc.) reference SDR images (e.g., a sequence of consecutive or sequential reference SDR images, etc.) decoded from the video signal to generate or reconstruct reshaped HDR images approximating reference HDR images that can or could be generated from the same upstream capture device that generate or capture the reference SDR images.

[0065] As TPB optimization is only used in the backward path, the resultant look of the reshaped HDR images generated from backward reshaping the reference SDR images may have relatively large deviations from a desired look of the reference HDR images in the BB1 operational scenarios.

[0066] In some BB1 operational scenarios, as illustrated in FIG. 1D, camera raw images generated from a camera ISP instead of reference SDR images generated from a programmable image signa processor may be directly encoded in the video signal outputted by the upstream (capture) device. In a training phase, training camera raw images, which may or may not be SDR images, can be spatially and/or temporally aligned with training HDR images int the same image pairs of camera raw and HDR images. Matching color pairs from the image pairs of spatially aligned camera raw and HDR images can be used to optimize backward reshaping operational parameters such as backward TPB coefficients. In a non-training or deployment phase, the optimized backward reshaping operational parameters can be used by a downstream device of a video signal encoded with non-training camera raw images to backward reshape decoded non-training camera raw images from the video signal into backward reshaped or reconstructed (non-training) HDR images that approximate reference HDR images that can or could be generated by the same upstream (capture) device.

[0067] FIG. 1E illustrates a fourth example reshaping optimization process implementing a black-box backward only reshaping optimization design/solution (referred to as BB2) with different video capture devices. The BB2 reshaping optimization design/solution may be implemented for operational scenarios in which SDR images are generated by a first capture device and HDR images to be generated from the SDR images are (e.g., intended to be, etc.) generated by a second different capture device and in which only the backward (reshaping) path is subjected to reshaping optimization. In addition, the reshaping optimization in the BB2 operational scenarios may or may not be TPB optimization.

[0068] Reshaping operational parameters in the BB2 operational scenarios can be generated with training SDR images and training HDR images generated or acquired respectively by the two video capture devices such as two mobile devices of different makes and/or models. These training SDR images and training HDR images form a plurality of SDR and HDR image pair each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

[0069] A training SDR image and a training HDR image in the same SDR and HDR image pair may be acquired using the same image such as the same checkerboard image rendered on the same or similar image display(s). In the BB1 operational scenarios, relationships between pixel or codeword values in the training SDR image and corresponding pixel or codeword values in the training HDR image in the same image pair may be treated as, or assumed to be, a black box.

[0070] Training SDR and HDR images in each of the image pairs may be first spatially aligned. Spatially aligned training SDR and HDR images in the image pairs may be used to determine or find out matching color patches rendered with test images such as checkerboard images on the same image display or the same image display type. These matching color patches can then be used to construct three-dimensional look-up tables (3D-LUTs) to map SDR pixel or codeword values of color patches in the test images to corresponding HDR pixel or codeword values of the same color patches.

[0071] The 3D-LUTs derived based in part or in whole on the test images can be used to generate optimized static or dynamic reshaping operational parameters for reshaping or mapping (e.g., non-training, reference, etc.) SDR images back to backward reshaped or reconstructed HDR images approximating (e.g., non-training, reference, etc.) HDR images.

[0072] As shown in FIG. 1E, SDR images such as reference SDR images may be generated by an upstream device such as a first capture device to be included/encoded in a video signalor a base layer (BL) thereofoutputted by the upstream device. Example reference SDR images as described herein may include, but are not necessarily limited to only, SDR images generated from a programmable image signal processor of the first capture device or upstream device.

[0073] A downstream recipient device or a video decoder of the video signal can decode the reference SDR images acquired with the first capture device from the video signal. The decoded reference SDR images at the decoder side may be the same as the reference SDR images at the encoder side, subject to errors introduced in compression, decompression, coding operations and/or data transmissions.

[0074] In the backward reshaping path as implemented by the downstream device, the reference SDR images may be backward reshaped into (backward) reshaped HDR images based at least in part on optimized backward reshaping operational parameters (denoted as Dynamic backward function optimization) generated from the BB2 reshaping optimization design/solution. The reshaped HDR imagesgenerated with the optimized backward reshaping operational parameters in an output HDR color spacerepresent an approximation to or reconstructed version of reference HDR images that can or could be generated by a second different capture device.

[0075] The BB2 reshaping optimization design/solution can generate the optimized backward reshaping operational parameters to cover as wide as possible in the output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

[0076] In some operational scenarios, the backward reshaping operational parameters generated from the BB2 reshaping optimization design/solution may be applied in a static or dynamic SLiDM framework.

[0077] As the BB2 reshaping optimization is only used in the backward path, the resultant look of the reshaped HDR images generated from backward reshaping the reference SDR images may have relatively large deviations from a desired look of the reference HDR images. The first and second capture devices may be mobile devices that can operate differently in their respective video capturing application, for example with different exposure times, different video processing operations, different mapping functions/relationships, etc.

[0078] A dynamic SLiDM frame in which dynamic mapping with image-dependent or image-specific reshaping operational parameters may be used to backward reshape SDR images of the first capture device to HDR images that can or could be generated by the second capture device, especially in operational scenarios in which the first and second capture devices operate with relatively large differences between respective video/image capture applications running on the first and second capture devices and/or between respective ISPs used in the first and second capture devices. For example, in a training phase, multiple sets of training SDR and HDR images or image pairs can be used to derive multiple sets of optimized reshaping operational parameters respectively for the multiple sets of training SDR and HDR images or image pairs. In a deployment or application phase, specific image characteristics such as overall brightness of some or all regions of a specific image can be dynamically evaluated when the specific image is being processed. A specific set of optimized reshaping operational parameters can be adaptively and/or dynamically selected for the specific image from among the multiple sets of optimized reshaping operational parameters based on the specific image characteristics of the specific image in relation to or in comparison with respective image characteristics of the multiple sets of training SDR and HDR images or image pairs.

White Box Joint TPB Forward and Backward Optimization (WFB)

[0079] A number of design factors may be considered for reshaping optimization to help increase coverage of a supported color space in which reshaped images are to be represented. First, tensor-product bi-spline (TPB) predictors with optimized TPB coefficients generated from reshaping optimization may be used to obtain or achieve a relative high prediction accuracy than other types of predictors including but not limited to MMR predictors. Multi-knot and continuity properties of tensor-product bi-spline predictors or prediction functions can be exploited or used to cover a relatively wide color range or color space portion with relatively high accuracy. Example TPB predictors can be found in U.S. Provisional Patent Application Ser. No. 62/908,770, Tensor-product B-spline predictor, by Guan-Ming Su, Harshad Kadu, Qing Song and Neeraj J. Gadgil, filed on Oct. 1, 2019, the contents of which are entirely incorporated herein by reference as if fully set forth herein.

[0080] In comparison, while multi-piece MMR predictors or prediction functions might be better than single-piece MMR predictors or prediction functions, discontinuity between different MMR pieces would likely be introduced with the multi-piece MMR predictors or prediction functions, thereby negatively impacting and even prohibiting usage of the multi-piece MMR predictors or prediction functions in many operational scenarios. Example MMR based operations are described in U.S. Pat. No. 8,811,490, which are incorporated by reference in its entirety as if fully set forth herein.

[0081] TPB predictors can be advantageously used in SLBC based video codec. More specifically, forward TPB predictors can be used to generate forward reshaped SDR images approximating reference SDR images, whereas backward TPB predictors can be used to generate backward reshaped or reconstructed HDR images approximating reference HDR images. Example usage of TPB predictors with SLBC based video codec can be found in U.S. Provisional Patent Application Ser. No. 63/255,057, Tensor-product B-spline prediction for HDR video in mobile applications, by H. Kadu et al., filed on Oct. 13, 2021, the contents of which are entirely incorporated herein by reference as if fully set forth herein. In addition, a BESA (Backward Error Subtraction for signal Adjustment) algorithm/method with modification to support neutral color preservation can be used in a pipeline of chained reshaping functions to optimize the forward and backward TPB predictors together to achieve a relatively high reversibility to or a relatively accurate approximation of the reference HDR images by the backward reshaped or reconstructed HDR images. Example BESA algorithm/method can be found in U.S. Provisional Patent Application Ser. No. 63/013,063, Reshaping functions for HDR imaging with continuity and reversibility constraints, by G-M. Su, filed on Apr. 21, 2020, U.S. Provisional Patent Application Ser. No. 63/013,807, Iterative optimization of reshaping functions in single-layer HDR image codec, by G-M. Su and H. Kadu, filed on Apr. 22, 2020, and PCT Application Ser. No. PCT/US2021/028475, filed on Apr. 21, 2021, the contents of which are entirely incorporated herein by reference as if fully set forth herein.

[0082] While TPB predictors have better prediction accuracy than other types of predictors, a relatively large signal or bitrate overhead may be incurred to transmit TPB coefficients in a video signal. In addition, a relatively large computational cost may be incurred to construct TPB equations or basis functions used in the TPB predictors.

[0083] In some operational scenarios, to avoid or reduce the signal or bitrate overhead and computational cost, built-in or static TPB predictors may be used to reshape some or all images in an entire video sequence without changing TPB coefficients used in the static TPB predictors during playing back the video sequence. More specifically, some or all of the (built-in or static) TPB coefficients can be cached or stored in a video application such as a video capture/editing application without needing to transmit these TPB coefficients explicitly through or in a video signal or bitstream encoded with images to be reshaped by the built-in or static TPB predictors. In an example, some or all of the TPB coefficients can be pre-loaded or preconfigured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. Additionally, optionally or alternatively, multiple sets of TPB coefficients can be pre-loaded or preconfigured in a downstream recipient device before the video signal or bitstream is received and processed by the downstream recipient device. The video applications can simply choose or select a specific set of TPB coefficients to use with the static TPB predictors from among the multiple sets of TPB coefficients, for example based on a simple indicator (e.g., a binary indicator, a multi-bit indicator, etc.) signaled or transmitted in the video signal or bitstream. Example static TPB predictors or prediction functions can be found in in the previously mentioned U.S. Provisional Patent Application Ser. No. 63/255,057.

[0084] In some operational scenarios, mobile devices may be used to host and run video capture and/or editing applications. Users of the mobile devices can edit captured images in these video capture and/or editing applications. The edited images such as edited HDR imageswhich, for example, may be intended to be displayed or viewed on image displays of non-mobile devices with display capabilities higher or larger than those of image displays of the mobile devicescan have luminance ranges and/or color ranges or color distributions higher, larger, wider and/or broader than those of the (original) captured images such as HDR images originally captured from camera sensors of the mobile devices. For example, the captured HDR images can often be limited by camera sensors and ISP outputs of the mobile devices to be represented in a relatively narrow color space or gamut such as P3, whereas the edited HDR images can be represented in a relatively wider color space or gamut such as the entire R.2020 color space.

[0085] To design a static mapping for reshaping operations, it is highly desired to optimize forward/backward TPB coefficients to cover dynamic range(s) as high as possible and color space(s) or gamut(s) as wide as possible while realizing bitrate and computational efficiency as much as possible.

[0086] A full HDR revertability between a reconstructed HDR image and a reference HDR image to which the reconstructed HDR is identical would need to distinguish colors in an original or reconstructed HDR domain (or color space) to be distinguished or distinguishable in a reshaped SDR domain (or color space), in order to enable the distinguished colors in the SDR domain to be mapped back to distinguishable colors in the HDR domain. Hence, the full revertability would likely need one-to-one mappings from HDR to SDR as well as from SDR back to HDR. The wider the HDR domain or color space is to support, the more codewords the SDR domain or color space needs to have. Given the total number of available SDR pixel or codeword values in the SDR domain (e.g., corresponding to a lower bit depth than that of the HDR domain, etc.) is typically smaller than the total number of needed HDR pixel or codeword values in the HDR domain, the full HDR revertability may not be possible in some operational scenarios. Indeed, some captured or edited images may contain combinations of diffusive and/or specular colors forming color distributions such as shown as an irregular shape in FIG. 2A that go beyond, and hence are not entirely representable by, the R.2020 color space (BT.2020), let alone a smaller color space or SDR color space such as the R.709 color space (BT.709) or the (DCI) P3 color space.

[0087] In many operational scenarios, a non-linear reshaping mapping or function can be used to distribute available pixel or codeword values relatively efficiently and generate the (forward) reshaped SDR that approximates the reference SDR image as closely as possible and that helps the reconstructed HDR image to approximate the reference HDR image as closely as possible, albeit the reshaped SDR generated with the non-linear reshaping mapping or function may not be identical to the reference SDR image but rather may contain some deviations from the reference SDR image.

[0088] Additionally, optionally or alternatively, in some operational scenarios, to maintain the same or similar looks of the reference SDR and HDR images in the reshaped SDR and HDR images, reshaping optimization as described herein can be performed with neutral color constraints that preserve neural colors in color mapping performed with the reshaping operations.

[0089] Given their capabilities to approximate functions with relatively high non-linearities, TPB predictors or functions can be incorporated in the reshaping operations to support or realize a relatively wide HDR color space for representing the backward reshaped or reconstructed HDR images.

[0090] A potential downside is that the TPB predictors may need a relatively large number of TPB coefficients to represent or approximate non-linear SDR-to-HDR and/or HDR-to-SDR mapping functions. In order to prevent possible ill-defined conditions, numerical instabilities, slow convergence issues, etc., that may occur in solving optimization problems for TPB coefficients, static TPB predictors may be generated beforehand and deployed with video codecs such as those used by upstream devices and/or downstream devices before these video codecs are used to process video sequences such as capture and/or edited video sequences.

[0091] For the purpose of illustration only, in the operational scenarios as shown in FIG. 1A, the (forward reshaped) SDR domain or color space may be that of R.709, whereas the (original or backward reshaped) HDR domain or color space may be that of R.2020. As illustrated in FIG. 2A, the R.2020 color space is much larger than the R.709 color space. Hence, while it may be difficult to map the entire R.2020 to R.709 and then map back R.709 to R.2020, reshaping optimization techniques as described herein can be used to optimize the coverage of the HDR color space for representing the backward reshaped or reconstructed HDR images as well as maintain the same or similar looks of the reference SDR and HDR images in the reshaped SDR and HDR images. These techniques can be used to concurrently optimize reshaping operational parameters such as TPB coefficient for both forward and backward reshaping operations.

[0092] While it may not be possible in all scenarios to cover the entire R.2020 color space, a subset or sub-spacee.g., a specific color gamut delineated as a triangle formed by color primaries in a color coordinate system, etc.in the R.2020 color space may be selected or chosen by the reshaping optimization operations to maintain the HDR reversibility to the reference HDR images and acceptable SDR approximation of the reference SDR images in the reshaped HDR and SDR images.

[0093] By way of illustration but not limitation, the subset or sub-space in the R.2020 color space (or HDR color space for representing the reshaped HDR images) may be defined or characterized by a specific white point and three specific color primaries (red, green, blue). The specific white point may be selected or fixed to be the D65 white point. There may exist much design freedom to select the three specific color primaries from many possible color primary combinations for the subset or sub-space in the R.2020 to be supported by the reshaping operations. The reshaping optimization techniques as described herein can be used to determine or select the specific color primaries for the subset or sub-space in the R.2020 to be optimized color primaries for realizing or reaching maximum perceptual quality and/or color coding efficiency and/or revertability between SDR and HDR images.

[0094] A Macadam ellipse represents a just noticeable color difference in that the HVS may not be able to distinguish color differences within the same ellipse. The Pointer's gamut may represent all diffusive colors that can be perceived by the HVS. In Macadam ellipse and Pointer's gamut, green colors are less important or less distinguishable/perceptible by the HVS than red and blue colors. Hence, in some operational scenarios, the specific optimized color primaries for the subset or sub-space in the R.2020 to represent the reshaped or reconstructed HDR images can be selected to cover the red and blue colors more than the green colors, especially when the one-to-one mapping relations cannot be supported by SDR-to-HDR and HDR-to-SDR mappings in the reshaping operations.

[0095] As used herein, color primaries may also be referred to as primary colors and may be used to define corners of a polygon such as a triangle representing a specific color space or gamut such as a standard-specified color space, a display-supported color space, a video-signal-supported color space, etc. For example, a standard color space with a standard-specified white point can be represented by a triangle whose corners are specified by three standard-specified color primaries in the CIExy color space coordinate system or CIExy chromaticity diagram. CIExy coordinates of the color primaries (red or R, green or G, blue or B) and white points respectively defining the R.709, P3 and R.2020 color spaces are specified in TABLE 1 below.

TABLE-US-00001 TABLE 1 R G B White point R.709 (0.6400 0.3300) (0.3000 0.6000) (0.1500 0.0600) (0.3127 0.3290) P3 (0.6800 0.3200) (0.2650 0.6900) (0.1500 0.0600) (0.3127 0.3290) R.2020 (0.7080 0.2920) (0.1700 0.7970) (0.1310 0.0460) (0.3127 0.3290)

[0096] CIExy coordinates of color primaries of each of the standard color spaces in TABLE 1 and the corresponding white point may be denoted as

[00001]$\left(R_x^{\left(c\right)},R_y^{\left(c\right)}\right),\left(G_x^{\left(c\right)},G_y^{\left(c\right)}\right),\left(B_x^{\left(c\right)},B_y^{\left(c\right)}\right)\mathrm{and}\left(W_x^{\left(c\right)},W_y^{\left(c\right)}\right),$

respectively, where (c) is the standard color space.

[0097] Hence, the P3 color space may be characterized by CIExy coordinates of the P3 color primaries and the P3 white point as follows:

[00002]$\left(R_x^{\left(P3\right)},R_y^{\left(P3\right)}\right),\left(G_x^{\left(P3\right)},G_y^{\left(P3\right)}\right),\left(B_x^{\left(P3\right)},B_y^{\left(P3\right)}\right),\left(W_x^{\left(P3\right)},W_y^{\left(P3\right)}\right).$

Similarly, the R.2020 color space may be characterized by CIExy coordinates of the R.2020 color primaries and the R.2020 white point as follows:

[00003]$\left(R_x^{\left(R2020\right)},R_y^{\left(R2020\right)}\right),\left(G_x^{\left(R2020\right)},G_y^{\left(R2020\right)}\right),\left(B_x^{\left(R2020\right)},B_y^{\left(R2020\right)}\right),\left(W_x^{\left(R2020\right)},W_y^{\left(R2020\right)}\right).$

[0098] Denote a backward reshaped HDR color space for representing backward reshaped or reconstructed HDR images as an (a) color space. Hence, CIExy coordinates of the color primaries and white point defining the (a) color space may be denoted as

[00004]$\left(R_x^{\left(a\right)},R_y^{\left(a\right)}\right),\left(G_x^{\left(a\right)},G_y^{\left(a\right)}\right),\left(B_x^{\left(a\right)},B_y^{\left(a\right)}\right)\mathrm{and}\left(W_x^{\left(a\right)},W_y^{\left(a\right)}\right),$

respectively.

[0099] As noted, green colors are less important or less distinguishable/perceptible by the HVS than red and blue colors. In some operational scenarios, the red and blue color primaries of the (a) color space may be selected to match those of the R.2020 color space, as follows:

[00005]$\begin{array}{cc}{R}_x^{\left(a\right)}=R_x^{\left(R2020\right)}& \left(1-1\right)\end{array}$$\begin{array}{cc}{R}_y^{\left(a\right)}=R_y^{\left(R2020\right)}& \left(1-2\right)\end{array}$$\begin{array}{cc}{B}_x^{\left(a\right)}=B_x^{\left(R2020\right)}& \left(2-1\right)\end{array}$$\begin{array}{cc}{B}_y^{\left(a\right)}=B_y^{\left(R2020\right)}& \left(2-2\right)\end{array}$

[0100] In addition, like all the R.709, P3 and R.2020 color spaces that are specified with the D65 white points, the (a) color space can also be specified with the D65 white point, as follows:

[00006]$\begin{array}{cc}\left(W_x^{\left(a\right)},W_y^{\left(a\right)}\right)=\left(0.31270.329\right)& \left(3\right)\end{array}$

[0101] The green color primary of the (a) color space can be selected along the line between the green color primary

[00007]$\left(G_x^{\left(P3\right)},G_y^{\left(P3\right)}\right)$

of the P3 color space and the green color primary

[00008]$\left(G_x^{\left(R2020\right)},G_y^{\left(R2020\right)}\right)$

of the R.2020 color space in the CIExy coordinate system or chromaticity diagram. Any point along the line between the two green color primaries of the P3 color space and the R.2020 color space can be represented as a linear combination of these two green color primaries with a weight factor a, as follows:

[00009]$\begin{array}{cc}{G}_x^{\left(a\right)}={\mathrm{aG}}_x^{\left(P3\right)}+\left(1-a\right)G_x^{\left(R2020\right)}& \left(4-1\right)\end{array}$$\begin{array}{cc}{G}_y^{\left(a\right)}={\mathrm{aG}}_y^{\left(P3\right)}+\left(1-a\right)G_y^{\left(R2020\right)}& \left(4-2\right)\end{array}$

[0102] Hence, the optimization problem of finding the maximum support from the (a) color space for the R.2020 color space can be simplified as a problem to select the weight factor a. When a=0, the (a) color space becomes the entire R.2020 color space. When a=1, as illustrated in FIG. 2B (where the (a) color space is denoted as TPB or current TPB covered color), the red and blue corners or red and blue color primaries of the (a) color space are the same as the red and blue corners or red and blue color primaries of the R.2020 color space, whereas the green corner or green color primary of the (a) color space is the same as the green corner or green color primary in the P3 color space. FIG. 2C through FIG. 2E illustrate three example (a) color spaces for a=0.9, 0.5 and 0.25, respectively.

Joint Color Space and TPB Optimization

[0103] As the range or coverage by the backward reshaped (a) color space over the R.2020 color space may be controlled by the parameter a, an overall TPB (based reshaping) optimization problem becomes how to optimize TPB coefficients in both forward and backward paths, such that (i) the forward reshaped SDR domain (or color space) for representing reshaped SDR images is close to a reference SDR domain (or color space) for representing reference SDR images, especially in neutral colors or color space portions which are more sensitive to the HVS than non-neutral colors or color space portions, and (ii) the backward reshaped or reconstructed HDR domain (or color space) for representing backward reshaped or reconstructed HDR images is as closely identical as possible to the reference HDR domain (or color space) for representing reference HDR images. Ideally, the backward reshaped or reconstructed HDR images are identical to the reference HDR images if a perfect reconstruction or full revertability can be realized.

[0104] As illustrated in FIG. 2B through FIG. 2E, to cover as large the R.2020 color space as possible by the backward reshaped or reconstructed HDR domain or color space, the TPB optimization problem comes down to finding or searching for the smallest possible value for the parameter a such that the above SDR and HDR quality conditions are satisfied.

[0105] FIG. 3A illustrates an example process flow for finding the smallest possible value for the parameter a for the purpose of maximizing SDR and HDR quality in reshaped SDR and HDR images. The process flow of FIG. 3A may iterate through a plurality of candidate values for the parameter a in an iteration order which may be a sequential order or a non-sequential order.

[0106] Block 302 comprises selecting a current (e.g., to be iterated, etc.) value for the parameter a to the next candidate value (e.g., initially the first candidate value, etc.) in the plurality of candidate values for the parameter a. Given a candidate backward reshaped HDR color space with the current value for the parameter a, optimized reshaping operational parameters such as optimized TPB coefficients can be obtained in one or more subsequent process flow blocks of FIG. 3A.

[0107] Block 304 comprises building samples points, or prepare two sampled data sets, in the candidate backward reshaped HDR color space. By way of example but not limitation, the candidate backward reshaped HDR color space may be a Hybrid Log Gamma (HLG) RGB color space (referred to as the (a) RGB color space HLG).

[0108] The first of the two sampled data set is a uniformly sampled data set of color patches. Each color patch in the uniformly sampled data set of color patches is characterized or represented by a respective RGB color (denoted as

[00010]${\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(u\right)})$

uniformly sampled from the (a) RGB color space HLG, which includes three dimensions denoted as R-axis, G-axis, B-axis for R, G and B component colors, respectively. Each axis or dimension of the (a) RGB color space HLG is normalized to a value range of [0, 1]) and sampled by N.sub.R, N.sub.G, and N.sub.B units or divisions, respectively. Here, each of N.sub.R, N.sub.G, and N.sub.B represents a positive integer greater than one (1). Hence, the total number of color patches or sampled data points in the uniformly sampled data set of color patches is given as follows:

[00011]$\begin{array}{cc}{N}_u=N_R{N}_G{N}_B& \left(4\right)\end{array}$

[0109] Each uniformly sampled data point or RGB color {dot over (v)}.sub.ijk.sup.(u) in the uniformly sampled data set of color patches is given as follows:

[00012]$\begin{array}{cc}\begin{array}{c}{\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(u\right)}=\left[\begin{array}{ccc}\frac{i}{N_R-1}& \frac{j}{N_G-1}& \frac{k}{N_B-1}\end{array}\right]\\ \mathrm{for}i\left[0,.Math.,N_R-1\right],j\left[0,.Math.,N_G-1\right],k\left[0,.Math.,N_B-1\right]\end{array}& \left(5\right)\end{array}$

[0110] For simplicity, (i, j, k) in expression (5) above can be vectorized or simply denoted as p. Correspondingly, the uniformly sampled data point or RGB color

[00013]${\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(u\right)}.$

can be simply denoted as

[00014]${\stackrel{.}{v}}_p^{\left(u\right)}.$

All N.sub.u nodes (each of which represents a unique combination of a specific R-axis unit/partition, a specific B-axis unit/partition, a specific G-axis unit/partition) can be grouped or collected into a vector/matrix as follows:

[00015]$\begin{array}{cc}{\stackrel{.}{V}}^{\left(u\right)}=\left[\begin{array}{c}{\stackrel{.}{v}}_0^{\left(u\right)}\\ .Math.\\ {\stackrel{.}{v}}_{N_u-1}^{\left(u\right)}\end{array}\right]& \left(6\right)\end{array}$

[0111] The second of the two sampled data set prepared or built in block 304 is a neutral color data set. This second data set includes a plurality of neutral colors or neutral color patches (also referred to as gray colors or gray color patches).

[0112] The second data set may be used to preserve input gray color patches in an input domain as output gray color patches in an output domain when the input gray color patches in the input domain are mapped or reshaped into the output gray color patches in reshaping operations as described herein. The input gray color patches in the input domain (or input color space) may be given increased weighting in the optimization problem as compared with other color patches to reduce the likelihood of these input gray color patches being mapped to non-gray color patches in the output domain (or output color space) by the reshaping operations.

[0113] The second data setthe gray color data set or the data set of gray colorsmay be prepared or built by uniformly sampling the R, G, B values along the line connecting between a first gray color (0, 0, 0) and a second gray color (1, 1, 1) in the RGB domain (e.g., the (a) RGB color space HLG, etc.), giving rise to N.sub.n nodes or gray color patches, as follows:

[00016]$\begin{array}{cc}\begin{array}{c}{\stackrel{.}{v}}_i^{\left(n\right)}=\left[\begin{array}{ccc}\frac{i}{N_n-1}& \frac{i}{N_n-1}& \frac{i}{N_n-1}\end{array}\right]\\ \mathrm{for}i\left[0,.Math.,N_n-1\right]\end{array}& \left(7\right)\end{array}$

[0114] All the N.sub.n nodes in the second data set can be grouped or collected into a neutral color vector/matrix as follows:

[00017]$\begin{array}{cc}{\stackrel{.Math.}{V}}^{\left(n\right)}=\left[\begin{array}{c}{\stackrel{.}{v}}_0^{\left(n\right)}\\ .Math.\\ {\stackrel{.}{v}}_{N_n-1}^{\left(n\right)}\end{array}\right]& \left(8\right)\end{array}$

[0115] The neutral color vector/matrix in expression (8) above can be repeated N.sub.t (a positive integer no less than one (1)) times to generate N.sub.nN.sub.t neutral color patches in the (now repeated) second data set, as follows:

[00018]$\begin{array}{cc}{\stackrel{.}{V}}^{\left(n\right)}=\left[\begin{array}{c}{\stackrel{.Math.}{V}}^{\left(n\right)}\\ .Math.\\ {\stackrel{.Math.}{V}}^{\left(n\right)}\end{array}\right]& \left(9\right)\end{array}$

[0116] The repetition of neutral colors in the repeated second data set of neutral colors increases weighting of the neutral or gray colors as compared with other colors. Accordingly, the neutral colors can be preserved more in the optimization problem than the other colors.

[0117] The first data set of (all sampled) colors and the second (repeated) data set of neutral colors in expressions (6) and (9) can be collected or placed together in a single combined vector/matrix, as follows:

[00019]$\begin{array}{cc}{\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}=\left[\begin{array}{c}{\stackrel{.}{V}}^{\left(u\right)}\\ {\stackrel{.}{V}}^{\left(n\right)}\end{array}\right]& \left(10\right)\end{array}$

[0118] The total number of vector/matrix elements (repeated and non-repeated color patches) in the combined vector/matrix

[00020]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

is N=N.sub.nN.sub.t+N.sub.u. Each vector/matrix element or color patch (row) in

[00021]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

in expression (10) above may be denoted as follows:

[00022]$\begin{array}{cc}{\tilde{v}}_i^{\left(a\right)}=\left[\begin{array}{ccc}{\tilde{v}}_{\mathrm{ir}}^{\left(a\right)}& {\tilde{v}}_{\mathrm{ih}}^{\left(a\right)}& {\tilde{v}}_{\mathrm{ib}}^{\left(a\right)}\end{array}\right]& \left(11\right)\end{array}$

[0119] Block 306 comprises converting color values of the color patches (rows) represented in the vector/matrix elements of the combined vector/matrix

[00023]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

in the (a) RGB color space (or the (a) RGB color space HLG) to corresponding color values in a standard-based R.2020 color space or an R.2020 RGB color space HLG.

[0120] The three red, green and blue primary colors in the (a) RGB color space HLG, which correspond to three corners or points of a triangle defining the (a) RGB color space HLG in the CIExy chromaticity diagram, can be converted from CIE xy values to to CIE XYZ values via the following equations:

[00024]$\begin{array}{cc}Y=1& \left(12-1\right)\end{array}$$\begin{array}{cc}X=\frac{Y}{\mathrm{xy}}& \left(12-2\right)\end{array}$$\begin{array}{cc}Z=\frac{Y}{x\left(1-x-y\right)}& \left(12-3\right)\end{array}$

[0121] Given

[00025]$\left(R_x^{\left(a\right)},R_y^{\left(a\right)}\right),\left(G_x^{\left(a\right)},G_y^{\left(a\right)}\right),\left(B_x^{\left(a\right)},B_y^{\left(a\right)}\right)$

as the red, green and blue primary colors in (x, y) or CIE xy values, the same red, green and blue primary colors in (X, Y, Z) or CIE XYZ values, respectively denoted as

[00026]$\left(R_X^{\left(a\right)},R_Y^{\left(a\right)},R_Z^{\left(a\right)}\right),\left(G_X^{\left(a\right)},G_Y^{\left(a\right)},G_Z^{\left(a\right)}\right),\left(B_X^{\left(a\right)},B_Y^{\left(a\right)},B_Z^{\left(a\right)}\right)$

can be obtained using the conversion equations in expressions (12) above. Similarly, given

[00027]$\left(W_x^{\left(a\right)},W_y^{\left(a\right)}\right)$

as the white point in (x, y) or CIE xy values, the same white point in (X, Y, Z) or CIE XYZ values, denoted as

[00028]$\left(W_X^{\left(a\right)},W_Y^{\left(a\right)},W_Z^{\left(a\right)}\right),$

can be obtained using the same conversion equations in expressions (12) above.

[0122] A 33 conversion matrix denoted as P.sup.(a).fwdarw.XYZ can be constructed to convert the color values of the color patches (rows) represented in the vector/matrix elements of the combined vector/matrix

[00029]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

in (x, y) or CIE xy values to corresponding (X, Y, Z) or CIE XYZ values, as follows

[00030]$\begin{array}{cc}{P}^{\left(a\right).fwdarw.\mathrm{XYZ}}={\left(W^{\left(a\right).fwdarw.\mathrm{XYZ}}{\stackrel{.}{P}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\right)}^T& \left(13\right)\end{array}$

where denotes elementwise dot multiplication between the two matrixes on the right hand side (RHS) in expression (13); and the two matrixes on the RHS in expression (13) can be constructed as follows:

[00031]$\begin{array}{cc}{\stackrel{.}{P}}^{\left(a\right).fwdarw.\mathrm{XYZ}}=\left[\begin{array}{ccc}{R}_X^{\left(a\right)}& R_Y^{\left(a\right)}& R_Z^{\left(a\right)}\\ G_X^{\left(a\right)}& G_Y^{\left(a\right)}& G_Z^{\left(a\right)}\\ B_X^{\left(a\right)}& B_Y^{\left(a\right)}& B_Z^{\left(a\right)}\end{array}\right]& \left(14\right)\end{array}$$\begin{array}{cc}{W}^{\left(a\right).fwdarw.\mathrm{XYZ}}={\left[\begin{array}{c}{\stackrel{.Math.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\\ {\stackrel{.Math.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\\ {\stackrel{.Math.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\end{array}\right]}^T& \left(15\right)\end{array}$$\mathrm{where}$$\begin{array}{cc}{\stackrel{.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}=\left[\begin{array}{ccc}{W}_X^{\left(a\right)}& W_Y^{\left(a\right)}& W_Z^{\left(a\right)}\end{array}\right]& \left(16-1\right)\end{array}$$\begin{array}{cc}{\stackrel{.Math.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}={{\stackrel{.}{w}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\left({\stackrel{.}{P}}^{\left(a\right).fwdarw.\mathrm{XYZ}}\right)}^{-1}& \left(16-2\right)\end{array}$

[0123] An overall 33 conversion matrix denoted as P.sup.(a).fwdarw.R2020 can be constructed to convert the color values of the color patches (rows) represented in the vector/matrix elements of the combined vector/matrix

[00032]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

in the a RGB color space HLG to corresponding color values in the R.2020 RGB color space HLG, as follows:

[00033]$\begin{array}{cc}{P}^{\left(a\right).fwdarw.R2020}=P^{\left(a\right).fwdarw.\mathrm{XYZ}}{P}^{\mathrm{XYZ}.fwdarw.R2020}& \left(17\right)\end{array}$

where P.sup.XYZ.fwdarw.R2020 denotes a 33 conversion matrix that converts XYZ color values to corresponding values in the R.2020 RGB color space HLG.

[0124] Hence, in block 306, the color values of the color patches (rows) represented in the vector/matrix elements of the combined vector/matrix

[00034]${\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}$

in the a RGB color space (or the (a) RGB color space HLG) can be converted to the corresponding color values in a standard-based R.2020 color space or an R.2020 RGB color space HLG, as follows:

[00035]$\begin{array}{cc}{\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}={\tilde{V}}_{\mathrm{RGB}}^{\left(a\right)}{P}^{\left(a\right).fwdarw.R2020}& \left(18\right)\end{array}$

[0125] Block 308 comprises converting color values of the color patches (rows) represented in the vector/matrix elements of the vector/matrix

[00036]${\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 RGB color space HLG to corresponding color values (denoted as

[00037]${\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)})$

in an R.2020 YCbCr color space HLG, as follows:

[00038]$\begin{array}{cc}{\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)}=f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R2020}\left({\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}\right)& \left(19\right)\end{array}$

where

[00039]$f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R2020}\left(\right)$

denotes a conversion function or matrix that converts the (R.2020 RGB) color values in the R.2020 RGB color space HLG to the corresponding (HDR R.2020 YCbCr) color values in the R.2020 YCbCr color space HLG.

[0126] Each color patch (row) in

[00040]${\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)}$

may be denoted as follows:

[00041]$\begin{array}{cc}{\stackrel{}{v}}_i^{\left(R2020\right)}=\left[\begin{array}{ccc}{\stackrel{}{v}}_{i,Y}^{\left(R2020\right)}& {\stackrel{}{v}}_{i,\mathrm{Cb}}^{\left(R2020\right)}& {\stackrel{}{v}}_{i,\mathrm{Cr}}^{\left(R2020\right)}\end{array}\right]& \left(20\right)\end{array}$

[0127] Block 310 comprises converting or content mapping color values of the color patches (rows) represented in the vector/matrix elements of the vector/matrix

[00042]${\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 RGB color space HLG to corresponding color values (denoted as

[00043]${\stackrel{}{S}}_{\mathrm{YCbCr}}^{\left(R709\right)})$

in an R.709 SDR YCbCr color space, as follows:

[00044]$\begin{array}{cc}{\tilde{S}}_{\mathrm{RGB}}^{\left(R709\right)}=f_{\mathrm{HLG}.fwdarw.\mathrm{W\_SDR}}\left({\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}\right)& \left(21\right)\end{array}$$\begin{array}{cc}{\tilde{S}}_{\mathrm{YCbCr}}^{\left(R709\right)}=f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R709}\left({\tilde{S}}_{\mathrm{RGB}}^{\left(R709\right)}\right)& \left(22\right)\end{array}$

where .sub.HLG.fwdarw.W_SDR( ) denotes a white box HDR-to-SDR mapping function that converts the HDR R.2020 RGB color values

[00045]${\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 RGB color space HLG to corresponding R.709 SDR RGB color values (denoted as

[00046]${\stackrel{}{S}}_{\mathrm{RGB}}^{\left(R709\right)})$

in the R.709 SDR RGB color space; and

[00047]$f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R709}\left(\right)$

denotes a conversion function or matrix that then converts the R.709 SDR RGB color values

[00048]${\tilde{S}}_{\mathrm{RGB}}^{\left(R709\right)}$

in the R.709 SDR RGB color space to the corresponding color values

[00049]${\stackrel{}{S}}_{\mathrm{YCbCr}}^{\left(R709\right)}$

in the R.709 SDR YCbCr color space. A non-limiting example of the white box HDR-to-SDR mapping function is described in BT.2390-10, High dynamic range television for production and international programme exchange, (November 2021), which is incorporated herein by reference in its entirety. Other examples of white-box HDR-to-SDR mapping functions may include, but are not necessarily limited to only, any of: standard based HDR-to-SDR conversion functions, proprietary HDR-to-SDR conversion functions, HDR-to-SDR conversion functions with linear and/or non-linear domains, HDR-to-SDR mapping functions using gamma functions, power functions, etc.

[0128] Each color patch (row) in

[00050]${\tilde{S}}_{\mathrm{YCbCr}}^{\left(R709\right)}$

may be denoted as follows:

[00051]$\begin{array}{cc}{\stackrel{}{s}}_i^{\left(R709\right)}=\left[\begin{array}{ccc}{\stackrel{}{s}}_{i,Y}^{\left(R709\right)}& {\stackrel{}{s}}_{i,\mathrm{Cb}}^{\left(R709\right)}& {\stackrel{}{s}}_{i,\mathrm{Cr}}^{\left(R709\right)}\end{array}\right]& \left(23\right)\end{array}$

[0129] Block 312 comprises converting or content mapping color values of the color patches (rows) represented in the vector/matrix elements of the vector/matrix

[00052]${\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 RGB color space HLG to corresponding color values (denoted as

[00053]${\tilde{R}}_{\mathrm{YCbCr}}^{\left(R2020\right)})$

in an R.2020 YCbCr color space PQ, as follows:

[00054]$\begin{array}{cc}{\tilde{R}}_{RGB}^{\left(R2020\right)}=f_{\mathrm{HLG}.fwdarw.\mathrm{PQ}}\left({\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}\right)& \left(24\right)\end{array}$$\begin{array}{cc}{\tilde{R}}_{\mathrm{YCbCr}}^{\left(R2020\right)}=f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R2020}\left({\tilde{R}}_{\mathrm{RGB}}^{\left(R2020\right)}\right)& \left(25\right)\end{array}$

where .sub.HLG.fwdarw.PQ( ) denotes an HLG-to-PQ conversion function that converts the HDR R.2020 RGB HLG color values

[00055]${\tilde{V}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 RGB color space HLG to corresponding HDR R.2020 RGB PQ color values (denoted as

[00056]${\tilde{R}}_{\mathrm{RGB}}^{\left(R2020\right)})$

in the R.2020 HDR RGB color space PQ, for example using transfer functions described in the previously mentioned SMPTE 2084 and Rec. ITU-R BT.2100; and

[00057]$f_{\mathrm{RGB}.fwdarw.\mathrm{YCbCr}}^{R2020}\left(\right)$

denotes a conversion function or matrix that then converts the HDR R.2020 RGB PQ color values

[00058]${\tilde{R}}_{\mathrm{RGB}}^{\left(R2020\right)}$

in the R.2020 HDR RGB color space PQ to the corresponding color values

[00059]${\tilde{S}}_{\mathrm{YCbCr}}^{\left(R709\right)}$

in the R.2020 YCbCr color space PQ, for example using transfer functions described in the previously mentioned SMPTE 2084 and Rec. ITU-R BT.2100.

[0130] Each color patch (row) in

[00060]${\tilde{R}}_{\mathrm{YCbCr}}^{\left(R2020\right)}$

may be denoted as follows:

[00061]$\begin{array}{cc}{\stackrel{}{r}}_i^{\left(R2020\right)}=\left[\begin{array}{ccc}{\tilde{r}}_{i,Y}^{\left(R2020\right)}& {\tilde{r}}_{i,\mathrm{Cb}}^{\left(R2020\right)}& {\tilde{r}}_{i,\mathrm{Cr}}^{\left(R2020\right)}\end{array}\right]& \left(26\right)\end{array}$

[0131] Block 314 comprises taking

[00062]${\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)},{\stackrel{}{S}}_{\mathrm{YCbCr}}^{\left(R709\right)},\mathrm{and}{\tilde{R}}_{\mathrm{YCbCr}}^{\left(R2020\right)}$

as input to a (TPB) BESA algorithm, which may be enhanced or modified with neutral color preservation, to obtain or generate forward and backward reshaping operational parameters such as forward and backward TPB coefficients.

[0132] A BESA algorithm is an iterative algorithm in which each current iteration may modify a reference SDR signal according to backward prediction errors measured or determined in a previous iteration.

[0133] A modified BESA algorithm as described herein may implement neutral color preservation and avoid modifying neutral color patches. To do so, a set of neutral color patch indexes that identify or correspond to neutral color patches may be generated as follows:

[00063]$\begin{array}{cc}=\left\{i|.Math.{\stackrel{}{v}}_{i,\mathrm{Cb}}^{\left(R2020\right)}-0.5.Math.\mathrm{and}.Math.{\stackrel{}{v}}_{i,\mathrm{Cr}}^{\left(R2020\right)}-0.5.Math.\right\}& \left(27\right)\end{array}$

where represents a neutral color value threshold that determines or specifies a maximum range of color deviation in absolute values within which color values qualify as neutral colors. Example values for the neutral color value threshold 5 may include, but are not necessarily limited to only, any of: 1/2048, 1/1024, etc.

[0134] Let ch denote a channel in (e.g., three channels of, etc.) the forward reshaped SDR domain or color space. Let F denote forward reshaping or forward path. Let B denote backward reshaping or backward path.

[0135] A per-channel forward generation matrix (also referred to as a design matrix) denoted as

[00064]$S_F^{\mathrm{ch}}$

can be used to predict forward reshaped SDR color patches as characterized by corresponding forward reshaped SDR codewords from input HDR color patches

[00065]${\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)}$

as characterized by corresponding input HDR codewords derived with expression (19). The forward generation matrix can be generated or pre-calculated from cross-channel forward TPB basic functions

[00066]$B_{F,0}^{\mathrm{ch}}\left(\right)\mathrm{through}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left(\right),$

stored or cached in computer memory, and fixed in all iterations for the forward path, as follows:

[00067]$\begin{array}{cc}{S}_F^{\mathrm{ch}}=\left[\begin{array}{cccc}{B}_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R2020\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R2020\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R2020\right)}\right)\\ B_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R2020\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R2020\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R2020\right)}\right)\\ .Math.& .Math.& & .Math.\\ B_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R2020\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R2020\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R2020\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R2020\right)}\right)\end{array}\right]& \left(28\right)\end{array}$

where the cross-channel forward TPB basis functions

[00068]$B_{F,0}^{ch}\left(\right)\mathrm{through}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left(\right)$

take or accept the color patches (or rows) in

[00069]${\tilde{V}}_{\mathrm{YCbCr}}^{\left(R2020\right)}$

as shown in expression (20) above as input parameters.

[0136] The forward reshaped SDR color patches may be predicted by multiplying the forward generation matrix in expression (28) with forward TPB coefficients. Example prediction reshaped codewords (which are equivalent or similar to color patches herein) with TPB coefficients are described in in the previously mentioned U.S. Provisional Patent Application Ser. No. 62/908,770.

[0137] The forward reshaped SDR color patches or codewords can be backward reshaped into backward reshaped HDR color patches or codewords, for example through TPB based backward reshaping operations.

[0138] In the backward path, at each iteration (e.g., k-th iteration, etc.) in the BESA algorithm, a per-channel backward generation matrix denoted as

[00070]$S_B^{\mathrm{ch},\left(k\right)}$

can be used to predict backward reshaped HDR color patches as characterized by corresponding backward reshaped HDR codewords from the forward reshaped SDR color patches as characterized by the corresponding forward reshaped SDR codewords derived in the forward path. A backward generation matrix for each iteration can be generated from cross-channel forward TPB basis functions

[00071]$B_{F,0}^{\mathrm{ch}}\left(\right)\mathrm{through}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left(\right),$

and not fixed in all iterations for the backward path, as follows:

[00072]$\begin{array}{cc}{S}_F^{\mathrm{ch},\left(k\right)}=\left[\begin{array}{cccc}{B}_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{0,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{0,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)\\ B_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)\\ .Math.& .Math.& & .Math.\\ B_{F,0}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& B_{F,1}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)& .Math.& B_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}\left({\tilde{v}}_{N-1,Y}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\tilde{v}}_{N-1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\right)\end{array}\right]& \left(29\right)\end{array}$

where

[00073]${\hat{s}}_{0,Y}^{\left(R709\right),\left(k\right)},{\hat{s}}_{0,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\hat{s}}_{0,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}\mathrm{through}{\hat{s}}_{N-1,Y}^{\left(R709\right),\left(k\right)},{\hat{s}}_{N-1,\mathrm{Cb}}^{\left(R709\right),\left(k\right)},{\hat{s}}_{N-1,\mathrm{Cr}}^{\left(R709\right),\left(k\right)}$

represent the forward reshaped SDR color patches or codewords.

[0139] The backward reshaped HDR color patches may be predicted by multiplying the backward generation matrix in expression (29) with backward TPB coefficients.

[0140] Backward prediction errors for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patches or codewords collected in a vector/matrix with reference HDR color patches or codewords in a per-channel backward observation vector/matrix derived from expression (25). The per-channel backward observation vector/matrix can be generated or pre-calculated from expression (25), stored or cached in computer memory, and fixed in all iterations, as follows:

[00074]$\begin{array}{cc}{r}_B^{\mathrm{ch}}=\left[\begin{array}{c}{\tilde{r}}_{0,\mathrm{ch}}^{\left(R2020\right)}\\ {\tilde{r}}_{1,\mathrm{ch}}^{\left(R2020\right)}\\ .Math.\\ {\tilde{r}}_{N-1,\mathrm{ch}}^{\left(R2020\right)}\end{array}\right]& \left(30\right)\end{array}$

[0141] A reference SDR signal or reference SDR color patches or codewords may be used as prediction targets for the forward reshaping operations to generate the forward reshaped SDR color patches or codewords to approximate the reference SDR color patches or codewords. In the BESA algorithm, the reference SDR color patches or codewords

[00075]$\left\{{\stackrel{}{s}}_{i,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\right\}$

where k denotes an iteration index starting from zero (0) as the first iteration in the BESA algorithmmay be initialized to the SDR color patches or codewords derived with expression (22) above for the very first iteration (k=0), and thereafter may be modified at the end of each iteration based in part or in whole on the prediction errors determined for the iteration. The modified reference SDR color patches or codewords may be used in the next iteration in the BESA algorithm as prediction targets for the forward reshaping operations to generate the forward reshaped SDR color patches or codewords to approximate the modified reference SDR color patches or codewords

[0142] Per-channel reference SDR color patches or codewords at iteration k may be used to construct a vector/matrix

[00076]$r_F^{\mathrm{ch},\left(k\right)}$

as follows:

[00077]$\begin{array}{cc}{r}_F^{\mathrm{ch},\left(k\right)}=\left[\begin{array}{c}{\tilde{s}}_{0,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\\ {\tilde{s}}_{1,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\\ .Math.\\ {\tilde{s}}_{N-1,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\end{array}\right]& \left(31\right)\end{array}$

[0143] Before the first iteration, the vector in expression (31) above is set to the original reference SDR signal represented by expressions (22) and (23), as follows:

[00078]$\begin{array}{cc}{r}_F^{\mathrm{ch},\left(0\right)}=\left[\begin{array}{c}{\tilde{s}}_{0,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\\ {\tilde{s}}_{1,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\\ .Math.\\ {\tilde{s}}_{N-1,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\end{array}\right]& \left(31\right)\end{array}$

[0144] At iteration k, optimized values for the (e.g., per-channel, etc.) forward TPB coefficients (denoted as

[00079]$m_F^{\mathrm{ch},\left(k\right)})$

can be generated via a least squared solution to an optimization problem that minimizes differences between the forward reshaped SDR color patches or codewords and the reference SDR color patches or codewords determined for the iteration, as follows:

[00080]$\begin{array}{cc}{m}_F^{ch,\left(k\right)}={\left({\left(S_F^{ch}\right)}^T\left(S_F^{ch}\right)\right)}^{-1}\left({\left(S_F^{ch}\right)}^T{r}_F^{ch,\left(k\right)}\right)& \left(33\right)\end{array}$

[0145] The predicted SDR color patches or codewords at iteration k for channel ch can be computed from the optimized values for the (e.g., per-channel, etc.) forward TPB coefficients, as follows:

[00081]$\begin{array}{cc}{\stackrel{}{r}}_F^{ch,\left(k\right)}=S_F^{ch}{m}_F^{ch,\left(k\right)}=\left[\begin{array}{c}{\hat{S}}_{0,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\\ {\hat{S}}_{1,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\\ .Math.\\ {\hat{S}}_{N-1,\mathrm{ch}}^{\left(R709\right),\left(k\right)}\end{array}\right]& \left(34\right)\end{array}$

[0146] At iteration k, optimized values for the (e.g., per-channel, etc.) backward TPB coefficients (denoted as

[00082]$m_B^{ch,\left(k\right)})$

can be generated via a least squared solution to an optimization problem that minimizes differences between the backward reshaped HDR color patches or codewords and the reference HDR color patches or codewords, the latter of which are fixed for all the iterations in the BESA algorithm, as follows:

[00083]$\begin{array}{cc}{m}_B^{ch,\left(k\right)}={\left({\left(S_B^{\mathrm{ch},\left(k\right)}\right)}^T\left(S_B^{\mathrm{ch},\left(k\right)}\right)\right)}^{-1}\left({\left(S_B^{\mathrm{ch},\left(k\right)}\right)}^T{r}_B^{\mathrm{ch}}\right)& \left(35\right)\end{array}$

[0147] The predicted HDR color patches or codewords at iteration k for channel ch can be computed from the optimized values for the (e.g., per-channel, etc.) backward TPB coefficients, as follows:

[00084]$\begin{array}{cc}{\stackrel{}{r}}_B^{ch,\left(k\right)}=S_B^{\mathrm{ch},\left(k\right)}{m}_B^{ch,\left(k\right)}=\left[\begin{array}{c}{\hat{v}}_{0,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\\ {\hat{v}}_{0,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\\ .Math.\\ {\hat{v}}_{N-1,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}\end{array}\right]& \left(36\right)\end{array}$

[0148] Compute backward prediction error and propagate error back to reference non-neutral SDR signal.

[0149] As noted, the backward prediction errors for each iteration in the BESA algorithm can be determined by comparing the backward reshaped HDR color patches or codewords with the reference HDR color patches or codewords derived from expression (25) the latter of which are fixed for all the iterations in the BESA algorithm.

[0150] Among the backward prediction errors, backward prediction errors for non-neutral colors (or non-gray colors) can be propagated back to update or modify the reference SDR color patches or codewords for these non-neutral colors (or non-gray colors). The modified SDR color patches or codewords for the non-neutral colors (or non-gray colors) can be combined with the (non-modified) reference SDR color patches or codewords for the neutral colors (or gray colors) to serve as prediction target for the forward path in the next or upcoming iteration in the BESA algorithm.

[0151] In some operational scenarios, the backward prediction error can be computed as differences between the original or reference HDR signal (or the reference HDR color patches/codewords therein) and a predicted HDR signal (or the backward reshaped HDR color patches/codewords therein) at iteration k, as follows:

[00085]$\begin{array}{cc}{e}_{i,\mathrm{ch}}^{\left(k\right)}={\stackrel{}{v}}_{i,\mathrm{ch}}^{\left(R2020\right),\left(k\right)}-{\tilde{r}}_{i,ch}^{\left(R2020\right)}& \left(37\right)\end{array}$

[0152] In response to determining that a color patch i is in the neutral color set , reference SDR codewords for the color patch in the Cb and Cr channels of the reference SDR signal for the upcoming iteration (k+1) can be set to gray color values such as 0.5, as follows:

[00086]$\begin{array}{cc}{\stackrel{}{s}}_{i,\mathrm{Cb}}^{\left(R709\right),\left(k+1\right)}=0.5\mathrm{and}{\stackrel{}{s}}_{i,\mathrm{Cb}}^{\left(R709\right),\left(k+1\right)}=0.5& \left(38\right)\end{array}$

[0153] Otherwise, in response to determining that a color patch i is not in the neutral color set , the reference SDR codewords for the color patch in the Cb and Cr channels of the reference SDR signal for the upcoming iteration (k+1) can be set to updated or modified values according to the backward prediction errors, as follows:

[00087]$\begin{array}{cc}f\left(e_{i,\mathrm{Ch}}^{\left(k\right)}\right)=\min \left\{\left(e_{i,\mathrm{Ch}}^{\left(k\right)}.Math.>\right)?{}^{ch}{e}_{i,\mathrm{ch}}^{\left(k\right)}:0,\right\}& \left(39\right)\end{array}$$\begin{array}{cc}{\stackrel{}{s}}_{0,\mathrm{ch}}^{\left(R709\right),\left(k+1\right)}={\stackrel{}{s}}_{0,\mathrm{ch}}^{\left(R709\right),\left(k\right)}+f\left(e_{i,\mathrm{ch}}^{\left(k\right)}\right)& \left(40\right)\end{array}$

where represents a backward prediction error threshold above which a backward prediction error is enabled to be propagated to update the color patch of a non-neutral color; .sup.ch represents a fixed or configurable learning rate to scale the backward prediction error; represents an upper bound or maximum allowed value to modify the reference codewords for the color patch of the non-neutral color. An example value of E may, but is not necessarily limited to only, 0.000005. An example value of .sup.ch may, but is not necessarily limited to only, 1. An example value of may, but is not necessarily limited to only, 2. (which means no limit as SDR and/or HDR codewords herein may be normalized to a normalized domain or range of 0 to 1).

[0154] The neutral color preservation as enforced with expressions (38) through (40) can be used to ensure neutral colors in the reference SDR signal for all the iterations in the BESA algorithm from deviating to or toward non-neutral color. As a result, gray level related perceptual qualities in reshaped images are improved.

[0155] The BESA algorithm may be iterated up to a total number of iterations. In some operational scenarios, the total number of iterations for the BESA algorithm may be specifically selected to balance color deviations in reshaped SDR and/or HDR images. For example, different total numbers of iterations in the BESA algorithm may generate forward and backward TPB coefficients that produce reshaped SDR images of different SDR looks and reshaped HDR images of different HDR looks. A total number of iterations such as ten, fifteen, etc., that generate reshaped SDR and HDR images of relatively high quality looks can be selected for the BESA algorithm.

[0156] The BESA algorithm with neutral color preservation can be performed for each candidate value for the parameter a. For example, in block 314, the BESA algorithm with neutral color preservation can be performed for the current candidate value for the parameter a.

[0157] Block 316 comprises determining whether the current candidate value for the parameter a is the last candidate value in the plurality of candidate values for the parameter a. If so, the process flow goes to block 318. Otherwise, the process flow goes back to block 302.

[0158] Block 318 comprises selecting an optimal or optimized value for the parameter a as well as computing (or simply selecting already computed) the optimized forward and backward TPB coefficients for the (a) RGB color space corresponding to the optimized value for the parameter a.

[0159] The selection of the optimized value for the parameter a can be formulated as an optimization problem to find (a specific value set for)

[00088]$\left\{a,m_{F,a}^{ch},m_{B,a}^{ch}\right\}$

such that (1) the (a) RGB color space represents the largest HDR color space (e.g., to cover the largest portion of R.2020 RGB color space, etc.) to achieve minimized HDR prediction errors denoted as

[00089]$D_{HDR}\left(r_B^{\mathrm{ch}},{\hat{r}}_B^{\mathrm{ch}}\right),$

and (2) forward reshaped SDR color patches or codewords have relatively small or acceptable SDR color deviations denoted as

[00090]$D_{SDR}\left(r_F^{\mathrm{ch}},{\hat{r}}_F^{\mathrm{ch}}\right),$

as follows:

[00091]$\begin{array}{cc}\underset{\left\{a,m_{F,a}^{ch},m_{B,a}^{ch}\right\}}{\arg \min }\left(w.Math.D_{HDR}\left(r_B^{ch},{\stackrel{}{r}}_B^{ch}\right)+\left(1-w\right).Math.D_{SDR}\left(r_F^{ch},{\stackrel{}{r}}_F^{ch}\right)\right)& \left(41\right)\end{array}$

where w is a weighting factor used to balance between HDR and SDR errors.

[0160] This optimization problem can be solved by checking results from each a RGB color space using its corresponding optimal or optimized TPB coefficients m.sub.F,a.sup.ch and m.sub.B,a.sup.ch. Among all possible or candidate a values, the smallest a value (corresponding to the largest color space coverage in the R.2020 color space) that can achieve minimized HDR errors with relatively small SDR errors. The HDR prediction errors

[00092]$D_{HDR}\left(r_B^{\mathrm{ch}},{\hat{r}}_B^{\mathrm{ch}}\right)$

and the SDR color deviations

[00093]$D_{SDR}\left(r_F^{\mathrm{ch}},{\hat{r}}_F^{\mathrm{ch}}\right)$

may be subjective (e.g., with input from human observers, etc.) or objective distortion functions (e.g., with no or little input from human observers, etc.), such as those measured with an image dataset from a database. Example distortion functions may include, but are not necessarily limited to only, any of: mean squared error (or deviation) or MSE, root mean squared error (or deviation) or RMSE, sum of absolute difference or SAD, peak signal-to-noise ratio or PSNR, structural similarity index (SSIM), etc.

[0161] FIG. 2F and FIG. 2G illustrate example percentile Cb and Cr values in the forward reshaped SDR domain or color space for different values of the parameter a. Different values of the parameter a determine different color space coverages of the backward reshaped HDR domain or color space in the R.2020 domain or color space, and hence impact differently reconstruction or generation of backward reshaped HDR images in the R.2020 domain or color space and result in different distortionse.g., as represented or measured with the distortion measures

[00094]$D_{HDR}\left(r_B^{\mathrm{ch}},{\hat{r}}_B^{\mathrm{ch}}\right)$

of the backward reshaped HDR images as compared with reference HDR images which the backward reshaped HDR images are to approximate.

[0162] The vertical axis of FIG. 2F represents 3 percentile values in the Cb and Cr channels in the forward reshaped SDR domain for different values (e.g., obtained with the BESA algorithm with neutral color preservation for ten iterations, etc.) for the parameter a value as represented by the horizontal axis of FIG. 2F. The lower the value for the parameter a is, the larger in the R.2020 color space the a color space covers. The 3 percentile values may be used explicitly or implicitly to represent minimum values in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The smaller the 3 percentile values are, the smaller the minimum values are and hence the more extended the codeword value ranges in the forward reshaped SDR (YCbCr) domain or color space are. As it can be seen in FIG. 2F, the lower the value for the parameter a is, the lower the 3 percentile values in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space are.

[0163] The vertical axis of FIG. 2G represents 97 percentile values in the Cb and Cr channels in the forward reshaped SDR domain for different values (e.g., obtained with the BESA algorithm with neutral color preservation for ten iterations, etc.) for the parameter a value as represented by the horizontal axis of FIG. 2G. The 97 percentile values may be used explicitly or implicitly to represent maximum values in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The larger the 97 percentile values are, the larger the maximum values are and hence the more extended the codeword value ranges in the forward reshaped SDR (YCbCr) domain or color space are. As it can be seen in FIG. 2G, the 97 percentile values in both the Cb and Cr channels of the forward reshaped SDR domain or color space maintain near constant values.

[0164] In some operational scenarios, in block 318, the image dataset used to determine distortions in the reshaped SDR and HDR images and select the optimized value for the parameter a based in part or in whole on the distortions may include video bitstreams acquired with one or more specific video capturing devices such as mobile phone(s) (e.g., supporting video coding profiles in SMPTE ST 2094, supporting Dolby Vision Profile 8.4, etc.). A non-limiting example of the optimized value for the parameter a may, but is not necessarily limited to only, 0.5, which determines the largest color space coverage in the R.2020 color space for the backward reshaped HDR domain or color space. As a wider backward reshaped color space (e.g., corresponding to a value less than the optimized value for the parameter a, etc.) is used or supported, reshaped SDR colors and codewords can start to deviate from reference SDR colors and codewords with relatively large distortions. This is because increasing the backward reshaped HDR domain or color space means squeezing distinguished forward reshaped SDR colors (or codewords) more tightly into the forward reshaped SDR domain or color space, leading to move the forward reshaped SDR colors from original 3D positions represented in reference SDR colors (or codewords) to be approximated by the forward reshaped SDR colors.

[0165] As noted, forward and backward TPB coefficient for TPB based reshaping operations can be determined for a given value such as the optimized value for the parameter a. These TPB coefficients can be multiplied in math equations with a forward or backward generation matrix constructed with TPB basis functions using input codewords as input parameters to generate forward or reshaped codewords, which may entail performing numerous computations involving computing TPB basis function values for each pixel in numerous pixels of an image.

[0166] In some operational scenarios, to speed up or reduce computations, forward and backward 3D-LUTs may be constructed from pre-computed TPB basis function values from sampled values and the optimized forward and backward TPB coefficients that are generated for the optimized value for the parameter a. The forward and backward 3D-LUTs or lookup nodes/entries therein may be pre-built, before deployed at runtime to process input images, and applied with relatively simple lookup operations in the forward and backward paths or corresponding forward and backward reshaping operations performed therein with respect to the input images at the runtime.

[0167] Denote the optimized value for the parameter a as a.sup.opt. Denote the corresponding optimal forward and backward TPB coefficients as

[00095]$m_F^{\mathrm{ch},\mathrm{opt}}\mathrm{and}{m_B^{\mathrm{ch},\mathrm{opt}}}_{}^{},$

respectively.

[0168] The forward 3D-LUT can be used to forward reshape input HDR colors (or codewords) in an input HDR domain or color space such as the R.2020 domain or color space (in which the a color space is contained) into forward reshaped SDR colors (or codewords) in the forward reshaped SDR domain or color space.

[0169] The a color space identified inside the input HDR domain or color space such as the R 2020 (container) domain or color space may be used to clip out HDR colors or codewords represented outside the a color space. The forward TPB coefficients can be applied to input HDR colors or codewords in the a color space identified inside the input HDR domain or color space such as the R 2020 (container) domain or color space to generate predicted or forward reshaped SDR colors or codewords for each lookup node or entry in the forward 3D-LUT. As a result, the forward 3D-LUT includes a plurality of lookup nodes or entries each of which maps or forward reshapes a respective input (cross-channel or three-channel) HDR color or codeword to a corresponding predicted or forward reshaped (cross-channel or three-channel) SDR color or codeword.

[0170] In some operational scenarios, a forward 3D-LUT construction process includes a first step in which a 3D uniformly sampling grid is prepared in the input HDR domain or color space such as the R 2020 (container) domain or color space.

[0171] By way of illustration but not limitation, the input HDR domain or color space the R.2020 YCbCr color space HLG, which includes three dimensions or channels, namely Y-axis, Cb-axis, Cr-axis. Input HDR values (denoted as

[00096]${\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(\mathrm{FL}\right),\left(R2020\right)})$

made up of normalized values in a value range of [0, 1] in each of the axes can be uniformly sampled in each dimension or channel, as follows:

[00097]$\begin{array}{cc}{\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(\mathrm{FL}\right),\left(R2020\right)}=\left[\frac{i}{N_Y-1}\frac{j}{N_{\mathrm{Cb}}-1}\frac{k}{N_{\mathrm{Cr}}-1}\right]& \left(42\right)\end{array}$$\mathrm{for}i\left[0,.Math.,N_Y-1\right],j\left[0,.Math.,N_{\mathrm{Cb}}-1\right],k\left[0,.Math.,N_{\mathrm{Cr}}-1\right]$

where total numbers of per-axis sampled values along YCbCr axes are denoted as N.sub.Y, N.sub.Cb, and N.sub.Cr, respectively. Hence, the combined total number of sampled values in all these axes are N.sub.u=N.sub.YN.sub.CbN.sub.Cr.

[0172] While valid values in the entire R.2020 YCbCr color space may be limited (not to cover the entire 3D cube of YCbCr values), the entire sampling value grid may be used to cover the entire 3D cube of YCbCr values in order to reduce the likelihood of codeword deviations caused by compression operations.

[0173] For simplicity, (i, j, k) may be vectorized or denoted asp. Hence, the sampled values

[00098]${\stackrel{.}{v}}_{\mathrm{ijk}}^{\left(\mathrm{FL}\right),\left(R2020\right)}$

in the R.2020 YCbCr may be denoted as

[00099]${\stackrel{.}{v}}_p^{\left(\mathrm{FL}\right),\left(R2020\right)}.$

A vector/matrix can be constructed by collecting all N.sub.u nodes together, as follows:

[00100]$\begin{array}{cc}{\stackrel{}{V}}_{\mathrm{YCbCr}}^{\left(FL\right),\left(R2020\right)}=\left[\begin{array}{c}{\stackrel{.}{v}}_0^{\left(\mathrm{FL}\right),\left(R2020\right)}\\ .Math.\\ {\stackrel{.}{v}}_{N_u-1}^{\left(\mathrm{FL}\right),\left(R2020\right)}\end{array}\right]& \left(43\right)\end{array}$

[0174] The forward 3D-LUT construction process includes a second step in which the sampled values in the R.2020 YCbCr color space HLG may be converted to corresponding values (or colors) in the R.2020 RGB color space HLG, as follows:

[00101]$\begin{array}{cc}{\stackrel{}{V}}_{RGB}^{\left(FL\right),\left(R2020\right)}=f_{\mathrm{YCbCr}.fwdarw.\mathrm{RGB}}^{R2020}\left({\stackrel{}{V}}_{\mathrm{YCbCr}}^{\left(FL\right),\left(R2020\right)}\right)& \left(44\right)\end{array}$

[0175] The forward 3D-LUT construction process includes a third step in which the converted values in the R.2020 RGB color space HLG may be converted to corresponding values (or colors) in the a RGB color space HLG corresponding to the optimized value (denoted as a.sup.opt) for the parameter a, as follows:

[00102]$\begin{array}{cc}{\tilde{V}}_{\mathrm{RGB}}^{\left(\mathrm{FL}\right),\left(a\right)}={\left(P^{\left(a^{\mathrm{opt}}\right).fwdarw.R2020}\right)}^{-1}{\stackrel{.}{V}}_{\mathrm{RGB}}^{\left(\mathrm{FL}\right),\left(R2020\right)}& \left(45\right)\end{array}$

[0176] The forward 3D-LUT construction process includes a fourth step in which the converted values in the (a) RGB color space HLG may be clipped as follows:

[00103]$\begin{array}{cc}{\stackrel{}{V}}_{RGB}^{\left(FL\right),\left(a\right)}=\mathrm{clip}3\left({\tilde{V}}_{RGB}^{\left(FL\right),\left(a\right)},0,1\right)& \left(46\right)\end{array}$

[0177] The forward 3D-LUT construction process includes a fifth step in which the clipped values in the a RGB color space HLG may be converted to corresponding values (or colors) in the R.2020 RGB color space HLG, as follows:

[00104]$\begin{array}{cc}{V}_{RGB}^{\left(FL\right),\left(R2020\right)}=P^{\left(a^{\mathrm{opt}}\right).fwdarw.R2020}{\stackrel{}{V}}_{RGB}^{\left(FL\right),\left(a\right)}& \left(47\right)\end{array}$

[0178] The forward 3D-LUT construction process includes a sixth step in which the values in the R.2020 RGB color space HLG derived with expression (47) above may be converted to corresponding values (or colors) in the R.2020 YCbCr color space HLG, as follows:

[00105]$\begin{array}{cc}{V}_{\mathrm{YcbCr}}^{\left(FL\right),\left(R2020\right)}=f_{RGB.fwdarw.\mathrm{YCbCr}}^{R2020}\left(V_{RGB}^{\left(FL\right),\left(R2020\right)}\right)& \left(48\right)\end{array}$

[0179] The forward 3D-LUT construction process includes a seventh step in which, for each lookup node/entry in the forward 3D-LUT, the optimized forward TPB coefficients are used with a forward generation matrix

[00106]$S_F^{ch}$

built with the input HDR colors or codewords (or codeword values) in the R.2020 YCbCr color space HLG derived with expression (48) above as input parameters to forward TPB basis functions to obtain the mapped or forward reshaped SDR colors or codewords (or codeword values)

[00107]${\hat{r}}_F^{\mathrm{ch},\mathrm{opt}},$

as follows:

[00108]$\begin{array}{cc}{\hat{r}}_F^{\mathrm{ch},\mathrm{opt}}=S_F^{ch}{m}_F^{\mathrm{ch},\mathrm{opt}}=\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}{\stackrel{}{s}}_{0,\mathrm{ch}}^{\left(R709\right),\mathrm{opt}}\\ {\stackrel{}{s}}_{1,\mathrm{ch}}^{\left(R709\right),\mathrm{opt}}\end{array}\\ .Math.\end{array}\\ {\stackrel{}{s}}_{N_u,1,\mathrm{ch}}^{\left(R709\right),\mathrm{opt}}\end{array}\right]& \left(49\right)\end{array}$

[0180] In expression (49) above, the forward generation matrix

[00109]$S_F^{ch}$

can be built or constructed using the input HDR colors or codewords (or codeword values)

[00110]$V_{YCbCr}^{\left(FL\right),\left(R2020\right)}$

as input to the forward TPB basis functions, as follows:

[00111]$\begin{array}{cc}{S}_F^{ch}=\left[\begin{array}{cccc}\begin{array}{c}{B}_{F,0}^{\mathrm{ch}}(v_{0,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{0,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{0,Cr}^{\left(FL\right),\left(R2020\right)})\end{array}& \begin{array}{c}{B}_{F,1}^{\mathrm{ch}}(v_{0,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{0,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{0,Cr}^{\left(FL\right),\left(R2020\right)})\end{array}& .Math.& \begin{array}{c}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}(v_{0,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{0,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{0,Cr}^{\left(FL\right),\left(R2020\right)})\end{array}\\ \begin{array}{c}{B}_{F,0}^{\mathrm{ch}}(v_{1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{1,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{1,Cr}^{\left(FL\right),\left(R2020\right)})\end{array}& \begin{array}{c}{B}_{F,1}^{\mathrm{ch}}(v_{1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{1,\mathrm{Cb}}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{1,\mathrm{Cr}}^{\left(FL\right),\left(R2020\right)})\end{array}& .Math.& \begin{array}{c}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}(v_{1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{1,\mathrm{Cb}}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{1,\mathrm{Cr}}^{\left(FL\right),\left(R2020\right)})\end{array}\\ .Math.& .Math.& & .Math.\\ \begin{array}{c}{B}_{F,0}^{\mathrm{ch}}(v_{N_{u}1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{N_{u}1,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{N_{u}1,Cr}^{\left(FL\right),\left(R2020\right)})\end{array}& \begin{array}{c}{B}_{F,1}^{\mathrm{ch}}(v_{N_{u}1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{N_{u}1,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{N_{u}1,\mathrm{Cr}}^{\left(FL\right),\left(R2020\right)})\end{array}& .Math.& \begin{array}{c}{B}_{F,D_F^{\mathrm{ch}}-1}^{\mathrm{ch}}(v_{N_{u}1,Y}^{\left(\mathrm{FL}\right),\left(R2020\right)},\\ v_{N_{u}1,Cb}^{\left(\mathrm{FL}\right),\left(R2020\right)},v_{N_{u}1,\mathrm{Cr}}^{\left(FL\right),\left(R2020\right)})\end{array}\end{array}\right]& \left(50\right)\end{array}$