METHOD FOR PROCESSING HIGH DYNAMIC RANGE (HDR) DATA FROM A NONLINEAR CAMERA

Abstract

A method for processing high dynamic range data from a nonlinear camera includes; generating an input image comprising a plurality of pixels, each pixel having an initial pixel value, wherein the initial pixel values are generated using a camera transition curve; generating a first lookup table representing a combination of an inverse function and a re-compression function, the first lookup table having input values and output values, wherein each input value is linked to one output value, the inverse function is the inverse of the camera transition curve, the re-compression function is a smooth and continuous function having a slope at each input value which is greater than or equal to a corresponding slope of the camera transition curve, the first lookup table is generated such that the inverse function precedes the re-compression function; and generating a first image by converting the initial pixel values using the first lookup table.

Claims

1. A method for processing High Dynamic Range (HDR) data from a nonlinear camera, the method comprising: generating an input image with the camera, the input image comprising a plurality of pixels, each pixel having an initial pixel value, wherein the initial pixel values are generated using a camera transition curve; generating a first lookup table representing a combination of an inverse function and a re-compression function, the first lookup table having input values and output values, wherein each input value is linked to one output value, the inverse function is the inverse of the camera transition curve, the re-compression function is a smooth and continuous function having a slope at each input value which is greater than or equal to a corresponding slope of the camera transition curve, the first lookup table is generated such that the inverse function precedes the re-compression function; and generating a first image by converting the initial pixel values using the first lookup table.

2. The method in accordance with claim 1, wherein the re-compression function is a logarithmic function
y=a*ld(x)+b, wherein a and b are selected such that $\frac{d}{\mathrm{dx}}.Math.a*\mathrm{ld}\left(x\right)+b>=\frac{d}{\mathrm{dx}}.Math.{\mathrm{camera}}_{—}.Math.{\mathrm{transition}}_{—}.Math.\mathrm{curve}\left(x\right),$ wherein a and b are coefficients, x is an initial pixel value, and y is an output value.

3. The method in accordance with claim 1, wherein the first lookup table also represents a clipping function which maps all input values below or equal to one or all input values below or equal to a predefined threshold value to a value of zero, wherein the clipping function precedes the inverse function.

4. The method in accordance with claim 1, wherein an initial histogram of the input image is calculated and the initial histogram is transformed into a modified histogram using the first lookup table.

5. The method in accordance with claim 4, wherein the initial histogram is calculated as a weighted sum of sub-histograms of several regions of the input image, wherein each sub-histogram is preferably given equal weight.

6. The method in accordance with claim 5, wherein the modified histogram is transformed to a reduced histogram comprising of fewer bins than the modified histogram and a second lookup table is generated, which represents a modified histogram equalization based on the reduced histogram.

7. The method in accordance with claim 6, wherein the modified histogram equalization comprises altering the bin counts of the modified histogram such that all bin counts are between a predefined maximum value and a predefined minimum value, and wherein the modified histogram equalization further comprises convolving the modified histogram with a low-pass filter, preferably after the alteration of the bin counts of the modified histogram.

8. The method in accordance with claim 6, wherein the modified histogram equalization comprises generating the reduced histogram as a cumulative histogram from the modified histogram, preferably after the convolution, wherein the reduced histogram comprises of fewer bins than the modified histogram and wherein a histogram equalization using the reduced histogram is mapped in the second lookup table.

9. The method in accordance with claim 1, wherein the camera is a red/clear imager (RCCC imager), wherein one of the input image and the first image comprise red and grey pixels, wherein a second image is extracted from the first image, wherein the second image comprises the red pixels of the first image, and a third image is generated from the first image, wherein the red pixels of the first image are replaced by interpolating the values of neighboring grey pixels of the first image.

10. The method in accordance with claim 9, wherein the interpolated values of the third image are subtracted from the corresponding values of the red pixels of the second image thereby generating a fourth image and the fourth image is mapped to a red to clear ratio image using a linear mapping, wherein the value range of each pixel in the red to clear ratio image is smaller than in the fourth image.

11. The method in accordance with claim 9, wherein the third image is mapped to a grey image using the second lookup table, wherein the value range of each pixel in the grey image is smaller than in the third image.

12. The method in accordance with claim 9, wherein a white balance factor is calculated based on the function $w_b=2^{{\log }_2.Math..Math.f-\frac{1}{\mathrm{ap}}.Math.\Sigma \left(a.Math..Math.{\log }_2.Math.\frac{r}{g}\right)},$ wherein w.sub.b is the white balance factor, f is 1/3, p is a pixelcount, r is the value of a red pixel of the first image, and g is the value of a grey pixel of the first image.

13. The method in accordance with claim 12, wherein the resolution of the red to clear ratio image is interpolated to be equal to the resolution of the grey image, wherein an interpolated red to clear ratio image is generated and wherein to each pixel of the interpolated red to clear ratio image a value y=a*ld(g.sub.new)+b is added, with g.sub.new being the value of a grey pixel of the grey image, thereby creating a first modified interpolated red image, wherein the value y is generated by transforming the grey image using the first lookup table, and wherein a logarithm of the white balance factor is added to each pixel value of the first modified interpolated red image, thereby creating a second modified interpolated red image, a third lookup table is created, which represents the function $y=2^{\frac{x-b}{a}}$ wherein the third lookup table is used to create a red image from the second modified red image, one third of each pixel value of the red image is subtracted from each corresponding pixel value of the grey image and the resulting pixel values are equally distributed to a blue color channel and a green color channel and the red image, the blue color channel and the green color channel are combined to create a color image.

14. A camera system comprising a camera and a processing unit, the camera system being adapted to generate an input image with the camera, the input image comprising a plurality of pixels, each pixel having an initial pixel value, wherein the initial pixel values are generated using a camera transition curve; generate a first lookup table representing a combination of an inverse function and a re-compression function, the first lookup table having input values and output values, wherein each input value is linked to one output value, the inverse function is the inverse of the camera transition curve, the re-compression function is a smooth and continuous function having a slope at each input value which is greater than or equal to a corresponding slope of the camera transition curve, the first lookup table is generated such that the inverse function precedes the re-compression function; and generate a first image by converting the initial pixel values using the first lookup table.

15. A vehicle with the camera system of claim 14, wherein output data of the camera system is used for a machine vision application.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0069] The present invention will exemplarily be explained in more detail in the following with reference to preferred embodiments and to the accompanying drawings in which are shown:

[0070] FIG. 1 is a block diagram showing a method in accordance with the present invention;

[0071] FIG. 2 is a block diagram showing the reconstruction of a color image;

[0072] FIG. 3 is a block diagram depicting the generation of a first lookup table; and

[0073] FIG. 4 is a camera image showing the definition of image regions along a driving path.

DETAILED DESCRIPTION

[0074] The method depicted in FIG. 1 starts with the acquisition of an input image 10 by a camera (not shown). The camera is mounted on a vehicle, wherein the input image shows a view through the windshield of the vehicle. The camera is a RCCC imager, thus, the input image comprises red and grey pixels. From the input image 10 an initial histogram 12 is calculated in box 14.

[0075] The calculation of the initial histogram 12 is shown in more detail in FIG. 4. FIG. 4 shows a road 15 which is depicted in the input image 10. From the input image 10 three different regions 16 are used to calculate respective sub-histograms 18. One region 16 comprises the whole of the input image 10, whereas the other regions 16 comprise only parts of the input image 10, which are aligned along the road 15. For a histogram of the whole of the input image 10, a histogram embedded in the imager data from the camera can be used.

[0076] Each sub-histogram is weighted with a factor w.sub.1, w.sub.2, w.sub.3, respectively, wherein the initial histogram 12 is the weighted sum of the sub-histograms 18. As an example, all weights w.sub.1, w.sub.2, w.sub.3 are equal.

[0077] Coming back to FIG. 1, a first lookup table 20 is generated in box 22, wherein the first lookup table 20 represents a function that comprises three calculation steps, as is shown in more detail in FIG. 3.

[0078] The first calculation step is a clipping function 24 (FIG. 3) that maps all values below or equal to 1 to a value of 0. All other values remain the same.

[0079] The second calculation step is an inverse function 26 which represents the inverse of a camera transition curve 28. The camera transition curve 28 is an attribute of the used camera.

[0080] The third calculation step is a logarithmic re-compression function 30. The re-compression function 30 is a smooth and continuous function and has a slope at each input value which is greater than a corresponding slope of the camera transition curve 28.

[0081] The first lookup table 20 is calculated such that it represents a subsequent execution of the clipping function 24, the inverse function 26, and the re-compression function 30.

[0082] The input image 10 can use e.g. 12 bits per pixel; consequently, the first lookup table 20 can comprise output values for 4096 input values. The mapping created by the first lookup table 20 maps the 12 bit data of the input image 10 to data using 15 bits per pixel.

[0083] Now referring to FIG. 1 again, the first lookup table 20 is used to map the initial histogram 12 to a modified histogram 32 (box 34). In box 20 the input image 10 is converted into a first image 36 using the first lookup table 20. The red pixels are extracted from the first image 36 thereby forming a second image 40 in box 38.

[0084] In box 42 a demosaicing and interpolation of the first image 36 takes place, wherein the red pixels of the first image 36 are replaced by interpolating the values of two neighboring grey pixels of the first image 36 using those two neighboring pixels with the lowest gradient in between. This interpolation results in a third image 44.

[0085] In box 46 the interpolated values of the third image 44 are subtracted from the corresponding values of the red pixels of the second image 40. Thereby, a fourth image 48 is generated. In box 50 the fourth image 48 is linearly mapped to 8 bit resulting in a red to clear ratio image 52.

[0086] In box 56 the modified histogram 32 is transformed to a reduced histogram 54 containing only 64 bins. The transformation of the modified histogram 32 to the reduced histogram 54 comprises an alteration of the bin counts of the modified histogram 32, a convolution of the modified histogram 32 with a low-pass filter and the generation of the reduced histogram 54 as a cumulative histogram of the modified histogram 32 (after the alteration and the convolution). Based on the reduced histogram 54, a second lookup table 58 is generated. The second lookup table 58 applies a modified histogram equalization thereby performing a mapping of data to 8 bits per pixel.

[0087] In box 60 the third image 44 is mapped to 8 bit using the second lookup table 58, thereby creating a grey image 62 (i.e., an LDR image—Low Dynamic Range image).

[0088] The method shown in FIG. 1 thus yields a grey image 62, a red to clear ratio image 52 and a modified histogram 32, which can be used for further processing in e.g. machine vision applications.

[0089] An exemplary further processing is shown in FIG. 2, wherein a color image is computed.

[0090] From the red to clear ratio contained in the red to clear ratio image 52 and/or from the first image 36, a white balance factor 64 is calculated in box 66 (FIG. 2), wherein the function

[00006]$\begin{array}{cc}{w}_b=2^{{\log }_2.Math..Math.f-\frac{1}{\mathrm{ap}}.Math.\Sigma \left(a.Math..Math.{\log }_2.Math.\frac{r}{g}\right)}& \mathrm{Eq}..Math.3\end{array}$

as introduced above is used.

[0091] In box 68 the resolution of the red to clear ratio image 52 is interpolated to be equal to the resolution of the grey image 62, wherein an interpolated red to clear ratio image 70 is generated.

[0092] In box 72 the grey image 62 is mapped to a logarithmic space using the function

y=a*ld(g.sub.new)+b  Eq. 4.

[0093] A resulting transformed grey image 73 is added to the interpolated red to clear ratio image 70, resulting in a first interpolated red image 76. A logarithmic value of the white balance factor 64 is added to each pixel value of the first interpolated red image 76, resulting in a second interpolated red image 78. The logarithmic value of the white balance factor 64 is calculated in box 74.

[0094] In box 80 a third lookup table is created which represents the function

[00007]$\begin{array}{cc}y=2^{\frac{x-b}{a}},& \mathrm{Eq}..Math.6\end{array}$

which is used to create a red image 81 from the second modified interpolated red image 78.

[0095] One third of each pixel value of the red image 81 is subtracted from each corresponding pixel value of the grey image 62 and the resulting pixel values are equally distributed, i.e., multiplied by a factor of 1.5, and outputted as a blue color channel 82 and a green color channel 84. The red image 81, the blue color channel 82, and the green color channel 84 are then combined to create a color image (not shown).

[0096] Since e.g. the first lookup table 20 and the third lookup table can be re-used for all images acquired with a certain camera, the lookup tables only have to be created once. The second lookup table 58 can be created histogram based and requires no modification of the image before. This results in a severe reduction of the required computational power to execute the described method.

[0097] Furthermore, due to the used logarithmic space values many complicated multiplication or division operations can be replaced with simpler additions or subtractions, thus creating an efficient method for processing high dynamic range data from a nonlinear camera.

[0098] While this invention has been described in terms of the preferred embodiments thereof, it is not intended to be so limited, but rather only to the extent set forth in the claims that follow.