FILTER STRENGTH CONTROL FOR ADAPTIVE LOOP FILTERING

Abstract

There are provided mechanisms for encoding an image. The method comprises determining adaptive loop filter, ALF, coefficient values. The method comprises determining a scaling factor. The method further comprises generating scaled ALF coefficient values by applying the scaling factor to one or more of the ALF coefficient values. The method comprises providing the scaled ALF coefficient values to a decoder, wherein providing the scaled ALF coefficient values to the decoder comprises encoding the scaled ALF coefficient values in a bitstream and conveying the bitstream over a network. The determined ALF coefficient values reduce an error between reconstructed image components and original image components and the determined scaling factor improves subjective performance for image to be encoded.

Claims

1. A method, performed by an encoder, for encoding an image, the method comprising: determining adaptive loop filter (ALF) coefficient values; determining a scaling factor; generating scaled ALF coefficient values by applying the scaling factor to one or more of the ALF coefficient values; and providing the scaled ALF coefficient values to a decoder, wherein providing the scaled ALF coefficient values to the decoder comprises encoding the scaled ALF coefficient values in a bitstream and conveying the bitstream over a network, wherein the determined ALF coefficient values reduce an error between reconstructed image components and original image components and wherein the determined scaling factor improves subjective performance for image to be encoded.

2. The method of claim 1, wherein applying the scaling factor to one or more of the ALF coefficient values comprises applying the scaling factor as a multiplication of the one or more of the ALF coefficient values in floating point representation or applying the scaling factor as a multiplication, addition, and/or shift of one or more filter coefficients in fixed point representation.

3. The method of claim 1, wherein determining the ALF coefficient values comprises solving a least-squares problem.

4. The method of claim 1, wherein determining the scaling factor comprises determining a strength of filtering with the ALF coefficient values.

5. The method of claim 4, wherein determining the strength of filtering with the ALF coefficient values comprises calculating one of: a sum of the absolute values of the ALF coefficient values, a sum of the squares of the ALF coefficient values or a square root of a sum of the squares of the ALF coefficient values.

6. The method of claim 4, wherein determining the scaling factor further comprises comparing the sum of the absolute values of the ALF coefficient values to 128.

7. The method of claim 4, wherein the strength of filtering with the ALF coefficient values is determined based on a quantization parameter (QP).

8. The method of claim 4, wherein determining the scaling factor further comprises comparing the determined strength of filtering with the ALF coefficient values to a strength threshold.

9. The method of claim 1, wherein determining the scaling factor comprises determining a classification type, wherein the classification type is one of vertical/horizontal, diagonal, and non-oriented, and the determined scaling factor is based on the determined classification type.

10. The method of claim 1, wherein the determined scaling factor is based on whether the image is an intra coded picture or an inter coded picture.

11. The method of claim 1, further comprising: determining that the scaling factor is not below 1; and using fixed ALF coefficient values only if the determined scaling factor is not below 1.

12. The method of claim 1, further comprising: quantizing the scaled ALF coefficient values; adjusting the quantized coefficient values; determining a strength of filtering with the scaled ALF coefficient values; and determining a strength of filtering with the adjusted quantized coefficient values, wherein the adjusted quantized coefficient values are such that the strength of filtering with the adjusted quantized coefficient values is not greater than the strength of filtering with the scaled ALF coefficient values by more than a threshold amount.

13. The method of claim 12, wherein adjusting the quantized coefficient values comprises only adjusting the quantized coefficient values if the determined scaling factor is less than 1.

14. The method of claim 1, wherein determining the scaling factor comprising determining how much adaptive loop filtering is used, and the scaling factor is determined based on how much adaptive loop filtering is used.

15. An apparatus adapted to: determine adaptive loop filter (ALF) coefficient values; determine a scaling factor; generate scaled ALF coefficient values by applying the scaling factor to one or more of the ALF coefficient values; and provide the scaled ALF coefficient values to a decoder, wherein providing the scaled ALF coefficient values to the decoder comprises encoding the scaled ALF coefficient values in a bitstream and conveying the bitstream over a network, wherein the determined ALF coefficient values reduce an error between reconstructed image components and original image components and wherein the determined scaling factor improves subjective performance for image to be encoded.

16. A method performed by a decoder for decoding an image, the method comprising: receiving scaled adaptive loop filter (ALF) coefficient values signaled by an encoder; reconstructing image components; and filtering the image components by applying the scaled ALF coefficient values to generate final reconstructed image components.

17. An apparatus (204) adapted to: receive scaled adaptive loop filter (ALF) coefficient values signaled by an encoder; reconstruct image components; and filter the image components by applying the scaled ALF coefficient values to generate final reconstructed image components, wherein the apparatus comprises a loop filter, wherein a portion of the loop filter is adapted to perform adaptive loop filtering in which the image components are reconstructed, and the scaled ALF coefficient values are applied.

18-20. (canceled)

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0094] FIG. 1 provides an illustration of coefficient reuse.

[0095] FIG. 2 illustrates a system comprising an encoder and a decoder according to some embodiments.

[0096] FIG. 3 illustrates an example encoder according to some embodiments.

[0097] FIG. 4 illustrates an example decoder according to some embodiments.

[0098] FIGS. 5A-5C and 6A-6C illustrate the effect of reducing filter strength on a sequence where ALF is used a lot.

[0099] FIG. 7 is a flow chart illustrating a process according to some embodiments.

[0100] FIG. 8 is a block diagram of an apparatus according to one embodiment.

DETAILED DESCRIPTION

[0101] System

[0102] FIG. 2 illustrates a system 200 according to an example embodiment. System 200 includes an encoder 202 and a decoder 204. In the example shown, decoder 204 receives, via a network 110 (e.g., the Internet or other network), encoded images produced by encoder 202.

[0103] FIG. 3 is a schematic block diagram of encoder 202. As illustrated in FIG. 3, the encoder 202 takes in an original image and subtracts a prediction 41 that is selected 51 from either previously decoded samples (“Intra Prediction” 49) or samples from previously decoded frames stored in the frame buffer 48 through a method called motion compensation 50. The task of finding the best motion compensation samples is typically called motion estimation 50 and involves comparing against the original samples. After subtracting the prediction 41 the resulting difference is transformed 42 and subsequently quantized 43. The quantized results are entropy encoded 44 resulting in bits that can be stored, transmitted or further processed. The output from the quantization 43 is also inversely quantized 45 followed by an inverse transform 46. Then the prediction from 51 is added 47 and the result is forwarded to both the intra prediction unit 49 and to the loop filter 100. The loop filter 100 may do deblocking, SAO, and/or ALF filtering. The result is stored in the frame buffer 48, which is used for future prediction. Not shown in the figure is that coding parameters for other blocks such as 42, 43, 49, 50, 51 and 100 also may also be entropy coded.

[0104] FIG. 4 is a corresponding schematic block diagram of decoder 204 according to some embodiments. The decoder 204 takes in entropy coded transform coefficients which are then decoded by decoder 61. The output of decoder 61 then undergoes inverse quantization 62 followed by inverse transform 63 to form a decoded residual. To this decoded residual, a prediction is added 64. The prediction is selected 68 from either a motion compensation unit 67 or from an intra prediction unit 66. After having added the prediction to the decoded residual 64, the samples can be forwarded for intra prediction of subsequent blocks. The samples are also forwarded to the loop filter 100, which may do deblocking, SAO processing, and/or adaptive ALF processing. The output of loop filter 100 is forwarded to the frame buffer 65, which can be used for motion compensation prediction of subsequently decoded images 67. The output of the loop filter 100 can also output the decoded images for viewing or subsequent processing outside the decoder. Not shown in the figure is that parameters for other blocks such as 63, 67, 66, and 100 may also be entropy decoded. As an example, the coefficients for the ALF filter in block 100 may be entropy decoded.

[0105] In some embodiments, the encoder 202 (e.g., the ALF portion of the loop filter 100 of the encoder 202) may be configured to determine ALF coefficient values, determine one or more scaling factors, and to generate scaled ALF coefficient values by applying the one or more scaling factors to the determined ALF coefficient values. The encoder 202 may signal the scaled ALF coefficient values to the decoder 204 (e.g., by encoding them into the video bit stream). The decoder 204 may be configured to use the scaled ALF coefficient values to decode an image. That is, the decoder 204 may be configured to reconstruct image components (e.g., to get Y.sub.SAO, Cb.sub.SAO, and Cr.sub.SAO) and obtain the scaled filter coefficients from the bit stream, and the decoder 204 (e.g., the ALF portion of the loop filter 100 of the decoder 204) may be configured to filter the image components by applying the scaled ALF coefficient values to generate the final reconstructed image components (e.g., Y.sub.ALF, Cb.sub.ALF, and Cr.sub.ALF).

[0106] In some embodiments, the system 200 may use the one or more scaling factors to control the strength of ALF filtering. In some embodiments, a scaling factor may attenuate or amplify the strength of the ALF filter. For example, in some embodiments, a scaling factor that is below 1 may attenuate the strength of the ALF filter, a scaling factor that is above 1 may amplify the strength of the ALF filter, and a scaling factor that is equal to 1 may keep the filter strength the same.

[0107] In some embodiments, the encoder 204 (e.g., the ALF portion of the loop filter 100 of the encoder 202) may apply a scaling factor as a multiplication of one or more filter coefficients in floating point representation (e.g., Cnew(i)=C(i)*sf, where C(i) are the filter coefficients for non-center neighboring samples, and sf is the scaling factor). In some alternative embodiments, the encoder 204 (e.g., the ALF portion of the loop filter 100 of the encoder 202) may apply a scaling factor as a multiplication, addition, and/or shift of one or more filter coefficients in fixed point representation (e.g., Cnew(i)=C(i)>>sfShift, where C(i) are the filter coefficients for non-center neighboring samples, and 2{circumflex over ( )}(−sfShift) is the scaling factor).

[0108] In some embodiments, determining one or more scaling factors may include determining one or more strengths of filtering (e.g., ALF filtering). In some embodiments, the encoder 202 may determine the strength of filtering as the sum of the absolute value of filter coefficients. In some alternative embodiments, the encoder 202 may determine the strength of filtering as the sum of the squares of the filter coefficients. In some other alternative embodiments, the encoder 202 may determine the strength of filtering as the square root of the sum of the squares of the filter coefficients. In some further alternative embodiments, the encoder 202 may determine the strength of filtering based on a quantization parameter (QP).

[0109] In some embodiments, the one or more scaling factors may be determined based on the one or more determined filter strengths. In some embodiments, determining the one or more scaling factors may include comparing the one or more determined strengths of filtering to one or more strength thresholds (e.g., a scaling factor that attenuates the filter strength may be used if the filter strength is above a strength threshold).

Filter Strength Calculation by Sum of Absolute Values (Example 1)

[0110] In some embodiments, the strength of ALF filtering is calculated by the sum of the absolute value of filter coefficients. In some embodiments, in floating point arithmetic, a sum of 1 may correspond to a filter that is un-restricted, a sum below 1 may correspond to a filter that is attenuated, and a sum above 1 may correspond to a filter that is amplified. In some fixed point arithmetic embodiments with a quantization of 1/128, a sum of 128 corresponds to a filter that is un-restricted, a sum below 128 corresponds to a filter that is attenuating, and a sum above 128 corresponds to a filter that has amplification.

Scaling Factors for Positive and Negative Filter Coefficients (Example 2)

[0111] In some embodiments, the encoder 202 may determine the strength of filtering separately for positive filter coefficients and negative filter coefficients. In some embodiments, the strength of filtering for positive filter coefficients may be the sum of positive filter coefficients, and the strength of filtering for negative filter coefficients may be the sum of negative filter coefficients.

[0112] Positive filter coefficients decrease the sample distance between the current sample and the neighboring samples with positive coefficients. Negative filter coefficients increase the sample distance between the current sample and the neighboring samples with negative coefficients.

[0113] In some embodiments, to control the strength of filtering, a first scaling factor may be used for positive filter coefficients, and a second scaling factor may be used for negative filter coefficients.

[0114] In some embodiments, a first scaling factor for positive filter coefficients that is equal to 1 and a second scaling factor for negative filter coefficients that is less than 1 may increase the low-pass effect of the filter. In some embodiments, a first scaling factor for positive filter coefficients that is equal to 1 and a second scaling factor for negative filter coefficients that is greater than 1 may increase the high-pass effect of the filter.

Fixed Filter Coefficients (Example 3)

[0115] In some embodiments where ALF allows for the use of fixed filter coefficients (e.g., pre-determined by the specification) that have varying filter strength, the use of fixed filter coefficients may be avoided when applying a scaling factor below 1. This may enable full control of the filter strength for ALF.

[0116] Some alternative embodiments may allow for usage of fixed filter coefficients that have a filter strength less than a threshold.

Fine-Tuning (Example 4)

[0117] In some embodiments, to enable control of the filter strength of ALF as part of filter optimization or filter selection, a maximum filter strength may be defined and kept track of during optimization and filter selection, and solutions that deviate too much from the maximum filter strength may be avoided.

[0118] In some embodiments, after filter coefficients have been quantized, the filter coefficients can be tuned to achieve better objective distortion and/or rate distortion by increasing or decreasing the filter coefficients from the quantized values. This adjustment of filter coefficients after quantization may be referred to as “fine-tuning”. By determining the filter strength before quantization and avoiding refinements in the fine-tuning stage that increase the filter strength more than a threshold, the strength of the filter can be maintained as part of the refinement. In some embodiments, refinement of filter coefficients after quantization may be omitted when the scaling factor is less than 1.

[0119] In some embodiments, this means that a condition may be added to the optimization so that the strength of the filter is kept constant during the optimization. As an example, this may mean that, if one coefficient is increased, another one must be decreased to arrive at a filter with the same strength. If the strength is measured as the sum of absolute values of the coefficients, this means that the sum of absolute values of the coefficients must be the same (or almost the same) during the optimization. If the strength is instead measured as the squared sum of absolute values of the coefficients, then the optimization instead must make sure that this measure does not change during optimization. This can be accomplished by minimizing the loss function:

[00002] $L = BDrate (d_{k}) + {λ [({.Math.}_{k = 0}^{12} c_{k}^{2}) - ({.Math.}_{k = 0}^{12} d_{k}^{2})]}^{2}$

[0120] Where c.sub.k is the coefficients before optimization and d.sub.k is a coefficient after optimization, and BDrate(d.sub.k) is the loss function that has traditionally been minimized. This can for instance be done by taking steps in the gradient of L,

[00003] $Δ d_{k} = - α \frac{\partial L}{\partial d_{k}},$

for a sufficiently small step size α such as α=0.001, and then update the coefficients using d.sub.k.sup.new=d.sub.k.sup.old+Δd.sub.k. The variable λ is used to set the balance between keeping the same strength (large λ) and between lowering the BD-rate penalty (small λ).

[0121] If instead the strength of the filter is measured as the sum of absolute values of the coefficients in the filter, the following loss function may instead be used:

[00004] $L = BDrate (d_{k}) + {λ [({.Math.}_{k = 0}^{12} .Math. c_{k}^{2} .Math.) - ({.Math.}_{k = 0}^{12} .Math. d_{k}^{2} .Math.)]}^{2} .$

[0122] Another way of ensuring the same strength is to test pairs of coefficients, where one coefficient is moved to increase the filter strength and the other coefficient is moved to decrease the filter strength by an equal amount or more. If the new combination results in a lower BDrate loss measure, the optimization procedure executes the change, and otherwise leaves it unchanged. It then proceeds to the next pair of coefficients.

[0123] In some alternative embodiments, after “optimal” filter coefficients have been obtained, the “optimal” filter coefficients may be merged such that fewer filters are defined due to rate distortion preference. Doing so can alter the filter strength of the merged filter compared to the filter strength of the “optimal” filter, and this can enable use of a stronger filter than necessary. By determining the filter strength before merging filters, mergings that results in filters that deviates too much from the “optimal” filter strengths may be avoided. In some embodiments, merging of filter coefficients may be omitted when the scaling factor is less than 1.

[0124] Some further alternative embodiments may include determining the optimal filter coefficients for a coding tree unit (CTU), determining the filter strength for respective filters, and then selecting filter coefficients (after merging, quantization, refinement) that maintain the determined filter strength and otherwise omit ALF for that CTU.

Quantization Parameter (QP) (Example 5)

[0125] In some embodiments, the filter strength of ALF may be controlled by the quantization parameter (QP). In some embodiments, a scaling factor less than 1 may be used to reduce the magnitude of the filter coefficients when QP is larger than a threshold (e.g., 36). In some embodiments, a filter strength larger than a QP dependent threshold is not allowed.

Scaling Factor Selection (Example 6)

[0126] In some embodiments, a scaling factor may be selected by testing different scaling factors and using the scaling factor that improves subjective performance for the content to be encoded. In some alternative embodiments, a scaling factor may be selected automatically by testing different scaling factors and selecting the scaling factor that maintains most of the objective performance of a non-scaled approach.

Filter Strength Control for Different Classification Types (Example 7)

[0127] In some embodiments, the filter strength can be controlled individually for different classification types (e.g., vertical/horizontal, diagonal, and non-oriented). In some embodiments, a specific scaling factor may be used for each of the respective classifications, and the scaling factors for the respective classifications may be different from one another.

Filter Strength Control for Intra vs. Inter Coded Pictures (Example 8)

[0128] In some embodiments, the filter strength may be controlled differently for intra coded pictures than for inter coded pictures. In some embodiments, only filter strengths equal to or less than a pre-defined filter strength may be allowed.

Filter Strength Calculation by Sum of Squares (Example 9)

[0129] In some embodiments, the filter strength may be controlled by measuring the sum of the squares of the filter coefficients. As an example, if c0=10, c1=0, c2=−20, c3 through c10=30, and c11=−40, then then sum of the squares of the filter coefficients is 10{circumflex over ( )}2+0{circumflex over ( )}2+(−20){circumflex over ( )}2+8*(30){circumflex over ( )}2+(−40){circumflex over ( )}2=9300.

[0130] In some embodiments, the encoder 202 may calculate a filter strength measure for all of the filters affecting an image. The maximum filter strength calculated for the filters affecting an image may be regarded as full strength. As an example, if there are four filters that affect an image, and their strength measures are 9300, 3200, 4600, and 10200, respectively, the encoder 202 may regard 10200 as full strength.

[0131] In some embodiments, the encoder 202 may apply a scaling factor to each of the filters that has a filter strength above a filter strength threshold. In some embodiments, the filter strength threshold may be a percentage or a fraction of the full strength measure. For example, a threshold factor of s=1.0 may result in no change to the filters because no filter will have a filter strength above the full strength measure (e.g., 10200×1.0). However, if the threshold factor is s=0.75, the encoder may make sure that no filter gets a strength larger than 0.75*10200=7650. This means that the filters with strength 3200 and 4600 will not be changed, but the filters with strength 9300 and 10200 will be scaled down until their strength is less than or equal to 7650. In some embodiments, scaling down the filters may be done by multiplying the filter coefficients with a factor r until the filter is below the strength threshold. In the example where c0=10, c1=0, c2=−20, c3 through c10=30, and c11=−40, scaling every filter coefficient with r=0.9 will give the new filter coefficients c0=9, c2=−18, c3 through c10=27, and c11=−36, and the strength after the scaling will become 7533, which is less than the filter strength threshold of 7650.

[0132] In some embodiments, the encoder 202 may determine the scaling factor r as:

r=sqrt(s*maxstrength/strength)=sqrt(0.75*10200/9300)=sqrt(0.8226)=0.9070.

[0133] In some embodiments, this scaling with a factor of r may be performed before quantization. However, this is not required, and, in some alternative embodiments, this scaling with a factor of r may be performed after quantization but before fine-tuning. In some additional embodiments, the scaling with a factor of r may instead be performed after fine tuning.

Filter Strength Calculation by Square Root of Sum of Squares (Example 10)

[0134] Some alternative embodiments may be similar to the embodiments using the sum of the squares of the filter coefficients as a measure for filter strength but may instead use the square root of the sum of squares of the filter coefficients as a measure for filter strength. As an example, if c0=10, c1=0, c2=−20, c3 through c10=30, and c11=−40, then the sum of the squares of the filter coefficients is 10{circumflex over ( )}2+0{circumflex over ( )}2+(−20){circumflex over ( )}2+8*(30){circumflex over ( )}2+(−40){circumflex over ( )}2=9300, and the measure is sqrt(9300)=96.44.

[0135] In some embodiments, the encoder 202 may calculate this measure for all the filters affecting an image. The maximum filter strength measure calculated for the filters affecting an image may be regarded as full strength. As an example, if there are four filters, and their strength measures are 96.44, 56.57, 67.82, and 100.99, respectively, the encoder 202 may regard 100.99 as full strength.

[0136] In some embodiments, the encoder 202 may apply a scaling factor to each of the filters that has a filter strength above a filter strength threshold. In some embodiments, the filter strength threshold may be a percentage or a fraction of the full strength measure. In some embodiments, the filter strength threshold may be equal to the full strength measure multiplied by a threshold factor s. For example, a threshold factor of s=1.0 may result in no change to the filters because none of the filters will have a filter strength above the full strength measure (e.g., 100.99×1.0). However, if the threshold factor is s=0.75, the encoder may make sure that no filter gets a strength larger than 0.75*100.99=75.75. This means that the filters with strength 56.57 and 67.8 will not be changed, but the filters with strength 96.44 and 100.99 will be scaled down until their strength is less than or equal to 75.75. In some embodiments, scaling down the filters may be done by multiplying the filter coefficients with a factor r until the filter is below the strength threshold. In the example where c0=10, c1=0, c2=−20, c3 through c10=30, and c11=−40, scaling every filter coefficient with r=0.7 will give the new filter coefficients c0=7, c2=−14, c3 through c10=21, and c11=−28, and the strength after the scaling will become 67.50, which is less than the filter strength threshold of 75.75 and will satisfy the criterion. However, scaling every filter coefficient with r=0.8 will give the new filter coefficients c0=8, c2=−16, c3 through c10=24, and c11=−32, and the strength after the scaling will become 77.14, which is higher than 75.75 and may not satisfy the criterion.

[0137] In some embodiments, the encoder 202 may determine the scaling factor r that perfectly hits the target as:

r=s*maxstrength/strength=0.75*sqrt(10200)/sqrt(9300)=sqrt(0.8226)=0.7855.

Because the coefficients may need to be rounded to integers afterwards, this calculated scaling factor r may still not hit the target exactly. In our example, multiplying with 0.7855 will give c0=7.854524, c1=0, c2=−15.709048, c3 through c10=23.563572 and c11=−31.418096. After rounding to integers we get c0=8, c1=0, c2=−16, c3 through c10=24 and c11=31, which gives the strength sqrt(5889)=76.74 which is too high. However, in some embodiments, the encoder 202 may nonetheless regard the scaled filter coefficients as close enough. Hence, in some embodiments, the encoder 202 may determine the filter coefficients by calculating the scaling factor r that perfectly hits the target, applying the calculated scaling factor to the filter coefficients, and then rounding each of the scaled filter coefficients to the closest integer.

Filter Strength Control Based on ALF Use (Example 11)

[0138] In some embodiments, the encoder 202 may control the filter strength based on how much ALF is used. In some embodiments, the encoder 202 may set a scaling factor that reduced the filter strength if ALF is used a lot.

[0139] Results: Visual Comparison

[0140] It may be hard to visualize video quality with a frame. However, as shown in FIGS. 5A-5C and 6A-6C, on sequences where ALF is used a lot (e.g., like on BasketballDrill), a reduction of the filter strength (e.g., by setting a scaling factor that is attenuates the filter strength) may reduce amount of blurring. FIGS. 5A-5C illustrate a BasketballDrill LDB at QP 37 and Picture Order Count (POC) 599. FIG. 5A-5C illustrate VTM-10.0, ALFstrength equal to 0.75, and ALFstrength equal to 0.5, respectively. FIG. 6A-6C illustrate the BasketballDrill LDB at QP 32 and POC 59. FIG. 6A-6C illustrate VTM-10.0, ALFstrength equal to 0.75, and ALFstrength equal to 0.5, respectively.

[0141] Results: Objective Comparison

[0142] Bjøntegaard delta rate results for Example 1 with a scaling factor of 0.75 when compared against VTM-10.0 are shown in the tables below. A figure of −1% means that it is possible to reach the same measured distortion with 1% less bits. The results indicate that the solution in embodiment 1 can be set to maintain the BDR of ALF but with less filter strength. Most of the objective benefit of ALF can be kept with a scaling factor of 0.75. In the tables below, values that have not yet been determining are indicated with a “TBD.”

[0143] The table below shows all intra over VTM-10.0:

TABLE-US-00003 Y U V Class A1 0.18% 0.49% 0.55% Class A2 TBD TBD TBD Class B 0.24% 0.17% 0.29% Class C 0.26% 0.18% 0.30% Class E 0.45% 0.24% 0.29% Overall TBD TBD TBD Class D 0.21% 0.30% 0.41% Class F TBD TBD TBD

[0144] The table below shows random access over VTM-10.0:

TABLE-US-00004 Y U V Class A1 0.31% 0.20% 0.34% Class A2 0.44% 0.18% 0.06% Class B 0.38% 0.19% 0.24% Class C 0.19% −0.12% −0.02% Class E Overall 0.33% 0.10% 0.16% Class D 0.22% −0.38% −0.20% Class F 0.22% 0.09% 0.18%

[0145] The table below shows low-delay B over VTM-10.0:

TABLE-US-00005 Y U V Class A1 Class A2 Class B TBD TBD TBD Class C 0.20% −0.06% −0.16% Class E 0.42% 0.30% 0.06% Overall TBD TBD TBD Class D 0.21% 0.23% −0.41% Class F 0.11% 0.67% 0.35%

[0146] Flowcharts

[0147] FIG. 7 illustrates a process 700 performed by the encoder 202 according to some embodiments. In some embodiments, the process 700 may be for encoding an image.

[0148] In some encoding embodiments, the process 700 may include a step 702 of determining ALF coefficient values. In some embodiments, the ALF coefficient values may be adaptive loop filter (ALF) coefficient values. In some embodiments, the ALF portion of the loop 100 may determine the ALF coefficient values.

[0149] In some embodiments, determining the ALF coefficient values in step 702 may include solving a least-squares problem. In some embodiments, the determined ALF coefficient values may reduce an error between reconstructed image components (e.g., Y.sub.SAO, Cb.sub.SAO, and Cr.sub.SAO) and original image components (e.g., Y.sub.org, Cb.sub.org, and Cr.sub.org).

[0150] In some encoding embodiments, the process 700 may include a step 704 of determining a scaling factor. In some embodiments, the determined scaling factor may improve subjective performance for image to be encoded. In some embodiments, the determined scaling factor may maintain most of the objective performance of a non-scaled approach.

[0151] In some embodiments, determining the scaling factor in step 704 may include determining a strength of filtering with the ALF coefficient values. In some embodiments, determining the strength of filtering with the ALF coefficient values may include calculating a sum of the absolute values of the ALF coefficient values. In some alternative embodiments, determining the strength of filtering with the ALF coefficient values may include calculating a sum of the squares of the ALF coefficient values. In some other alternative embodiments, determining the strength of filtering with the ALF coefficient values may include calculating a square root of a sum of the squares of the ALF coefficient values. In some embodiments, the strength of filtering with the ALF coefficient values may be determined based on a quantization parameter (QP).

[0152] In some embodiments, the scaling factor may be determined in step 704 based on the strength of filtering with the ALF coefficient values. In some embodiments, determining the scaling factor in step 704 may include comparing the determined strength of filtering with the ALF coefficient values to a strength threshold (e.g., comparing the sum of the absolute values of the ALF coefficient values to a strength threshold of 128 or comparing the QP to a strength threshold of 36). In some embodiments, determining the scaling factor in step 704 may include determining that the strength of filtering with the ALF coefficient values is larger than a strength threshold and, if the strength of filtering with the ALF coefficient values is determined to be larger than the strength threshold, setting the scaling factor such that a strength of filtering with the scaled ALF coefficient values is less than or equal to the strength threshold. In some embodiments, determining the scaling factor in step 704 may include determining that the strength of filtering with the ALF coefficient values is larger than a strength threshold and, if the strength of filtering with the ALF coefficient values is determined to be larger than the strength threshold, setting the scaling factor to be less than 1.

[0153] In some embodiments, the process 700 (e.g., step 704 of the process 700) may include determining the strength threshold to which the determined strength of filtering with the ALF coefficient values is compared. In some embodiments, determining the strength threshold may include (i) for each filter affecting the image, determining a strength of filtering with ALF coefficient values of the filter and (ii) multiplying a threshold factor by a maximum determined strength of the filters affecting the image. In some embodiments (e.g., embodiments where the determined strength of filtering with the ALF coefficient values is the calculated sum of the squares of the ALF coefficient values), the determined scaling factor may be equal to sqrt(s*maxstrength/strength), where sqrt is the square root, s is the threshold factor, maxstrength is the maximum determined strength of the filters affecting the image, and strength is the determined strength of filtering with the ALF coefficient values. In some embodiments (e.g., embodiments where the determined strength of filtering with the ALF coefficient values is the calculated square root of the sum of the squares of the ALF coefficient values), the determined scaling factor may be equal to s*maxstrength/strength, where s is the threshold factor, maxstrength is the maximum determined strength of the filters affecting the image, and strength is the determined strength of filtering with the ALF coefficient values.

[0154] In some embodiments, determining the scaling factor in step 704 may include determining a classification type (e.g., one of vertical/horizontal, diagonal, and non-oriented), and the determined scaling factor may be based on the determined classification type. In some embodiments, the scaling factor determined in step 704 may additionally or alternatively be based on whether the image is an intra coded picture or an inter coded picture.

[0155] In some embodiments, the process 700 may include steps of determining that the scaling factor is not below 1 and using fixed ALF coefficient values only if the determined scaling factor is not below 1. In some embodiments, the process 700 may include steps of determining that a strength of filtering with fixed filter coefficients is less than a threshold and using the fixed ALF coefficient values only if the determined strength is less than the threshold.

[0156] In some encoding embodiments, the process 700 may include a step 706 of generating scaled ALF coefficient values. In some embodiments, generating the scaled ALF coefficient values may include applying the scaling factor to one or more of the ALF coefficient values, and the scaled ALF coefficient values may be for use by the decoder 204 in filtering image components.

[0157] In some embodiments, applying the scaling factor to one or more of the ALF coefficient values in step 706 may include applying the scaling factor as a multiplication of the one or more of the ALF coefficient values in floating point representation. In some alternative embodiments, applying the scaling factor to one or more of the ALF coefficient values in step 706 may include applying the scaling factor as a multiplication, addition, and/or shift of one or more filter coefficients in fixed point representation.

[0158] In some embodiments, the process 700 (e.g., the step 706 of the process 700) may include rounding the scaled coefficients to the closest integer, and the rounded scaled coefficients may be used by the decoder 204 to in filtering image components.

[0159] In some embodiments, the process 700 may include an optional step 708 of quantizing the scaled ALF coefficient values. In some embodiments, ALF coefficient values and cross-component (CC) ALF coefficient values may be derived in floating point. In some embodiments where ALF coefficient values are derived in floating point, the ALF coefficient values in floating point may be quantized by multiplying them by 128 and then rounding them to integer values, the max value may be 128, and the min value may be −128. In some embodiments where CC ALF coefficient values are derived in floating point, the CC ALF filter coefficients may be quantized by representing them in multiples of 2, the max value may be 64, and the min value may be −64.

[0160] In some embodiments, the optional step 708 may additionally include adjusting the quantized coefficient values. In some embodiments, the step 708 may include determining a strength of filtering with the scaled ALF coefficient values and determining a strength of filtering with the adjusted quantized coefficient values. In some embodiments, the adjusted quantized coefficient values may be such that the strength of filtering with the adjusted quantized coefficient values are not greater than the strength of filtering with the scaled ALF coefficient values by more than a threshold amount. In some embodiments, adjusting the quantized coefficient values may include only adjusting the quantized coefficient values if the determined scaling factor is less than 1. In some embodiments, the optional step 708 may include determining optimal filter coefficients for a coding tree unit (CTU) and determining a strength of filtering with the optimal ALF coefficient values, and the adjusted quantized coefficient values may maintain the determined strength of filtering with the optimal ALF coefficient values.

[0161] In some embodiments, the process 700 may include an optional step 710 of providing the scaled ALF coefficient values to the decoder 204. In some embodiments, the scaled ALF coefficient values provided to the decoder 204 may be the quantized coefficient values (or the adjusted quantized coefficient values). In some embodiments, providing the scaled ALF coefficient values to the decoder 204 may include encoding the scaled ALF coefficient values in a bitstream and conveying the bitstream over the network 110.

[0162] In some embodiments, the process 700 may include avoiding merging of ALF coefficient values when the scaling factor is less than 1.

[0163] In some embodiments, the scaling factor determined in step 704 may be a first scaling factor, the one or more of the ALF coefficient values to which the first scaling factor is applied in step 706 may be a first set of the ALF coefficient values, the process 700 (e.g., the step 704 of the process 700) may include determining a second scaling factor, and generating the scaled ALF coefficient values in step 706 may include applying the second scaling factor to one or more of the ALF coefficient values in a second set of the ALF coefficient values. In some embodiments, the first set of the ALF coefficient values may be positive filter coefficients, and the second set of the ALF coefficient values may be negative ALF coefficient values. In some embodiments, determining the first scaling factor may include determining a first strength of filtering, and determining the second scaling factor may include determining a second strength of filtering. In some embodiments, determining the first strength of filtering may include calculating a sum of the first set of the ALF coefficient values, and determining the second strength of filtering may include calculating a sum of the second set of the ALF coefficient values. In some embodiments, the first scaling factor may be determined based on the first strength of filtering, and the second scaling factor may be determined based on the second strength of filtering.

[0164] FIG. 9 illustrates a process 900 performed by the decoder 204 according to some embodiments. In some embodiments, the process 900 may be for decoding an image. In some embodiments, the process 900 may include a step 902 of receiving scaled ALF coefficient values signaled by the encoder 202. In some embodiments, the process 900 may include a step 904 of reconstructing image components (e.g., to get Y.sub.SAO, Cb.sub.SAO, and Cr.sub.SAO). In some embodiments, the process 900 may include a step 906 of filtering the image components by applying the scaled ALF coefficient values to generate final reconstructed image components (e.g., Y.sub.ALF, Cb.sub.ALF, and Cr.sub.ALF). In some embodiments, a portion of the loop filter 100 of the decoder 204 may be adapted to perform adaptive loop filtering in which the image components are reconstructed, and the scaled ALF coefficient values are applied.

[0165] FIG. 8 is a block diagram of an apparatus 801 for implementing the encoder 202 or the decoder 204 according to some embodiments. That is, apparatus 801 can be adapted to perform the methods disclosed herein. In embodiments where the apparatus 801 implements the encoder 202, the apparatus 801 may be referred to as “encoding apparatus 801,” and, in embodiments where the apparatus 801 implements the decoder 204, the apparatus 801 may be referred to as a “decoding apparatus 801.” As shown in FIG. 8, the apparatus 801 may comprise: processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., one or more general purpose microprocessors and/or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed; one or more network interfaces 848 (which may be co-located or geographically distributed) where each network interface includes a transmitter (Tx) 845 and a receiver (Rx) 847 for enabling apparatus 801 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 848 is connected; and one or more storage units (a.k.a., “data storage systems”) 808 which may be co-located or geographically distributed and which may include one or more non-volatile storage devices and/or one or more volatile storage devices. In embodiments where PC 802 includes a programmable processor, a computer program product (CPP) 841 may be provided. CPP 841 includes a computer readable medium (CRM) 842 storing a computer program (CP) 843 comprising computer readable instructions (CRI) 844. CRM 842 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 844 of computer program 843 is adapted such that when executed by PC 802, the CRI causes apparatus 801 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 801 may be adapted to perform steps described herein without the need for code. That is, for example, PC 802 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software.

[0166] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0167] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

FILTER STRENGTH CONTROL FOR ADAPTIVE LOOP FILTERING

Assignee

Inventors

Cpc classification

Classification Explorer

H04N19/147

ELECTRICITY

Classification Explorer

H04N19/65

ELECTRICITY

Classification Explorer

H04N19/103

ELECTRICITY

Classification Explorer

H04N19/117

ELECTRICITY

Classification Explorer

H04N19/30

ELECTRICITY

Classification Explorer

H04N19/136

ELECTRICITY

Classification Explorer

H04N19/124

ELECTRICITY

International classification

Classification Explorer

H04N19/117

ELECTRICITY

Classification Explorer

H04N19/124

ELECTRICITY

Classification Explorer

H04N19/136

ELECTRICITY

Classification Explorer

H04N19/103

ELECTRICITY

Classification Explorer

H04N19/65

ELECTRICITY

Classification Explorer

H04N19/30

ELECTRICITY

Classification Explorer

H04N19/147

ELECTRICITY

Abstract

Claims

Description