Image processing device that generates an image from pixels with different exposure times

Abstract

An image processing apparatus includes an intermediate image generating unit configured to input an image which has been shot with differing exposure times set by region, generates a plurality of exposure pattern images corresponding to differing exposure times based on the input image, and generates a plurality of timing images which are difference images of the plurality of exposure pattern images; and a distortion correction processing unit configured to generate a corrected image equivalent to an exposure processing image at a predetermined exposure time by synthesizing processing of the plurality of timing images.

Claims

1. An imaging system, comprising: an imaging apparatus configured to have a plurality of pixels which are arranged in a plurality of pixel groups, wherein each pixel group of the plurality of pixel groups includes at least two pixels; and one or more processors configured to: generate a plurality of exposure pattern images based on each pixel signal from each pixel group of the plurality of pixel groups, wherein each pixel group of the plurality of pixel groups has different exposure times, and wherein the plurality of exposure pattern images correspond to the different exposure times; and generate a corrected image based on synthesis of the plurality of exposure pattern images that have different exposure start times within a set interval.

2. The imaging system according to claim 1, wherein each pixel of the plurality of pixels includes a photoelectric conversion element.

3. The imaging system according to claim 1, wherein each pixel of the plurality of pixels includes a transistor element.

4. The imaging system according to claim 3, wherein the transistor element is an amplifying transistor.

5. The imaging system according to claim 3, wherein the transistor element is a select transistor.

6. The imaging system according to claim 1, wherein the one or more processors are further configured to: generate an image based on each pixel group of the plurality of pixel groups; input the image which has been shot with different exposure times set by region; and generate the plurality of exposure pattern images that correspond to the different exposure times based on the input image.

7. The imaging system according to claim 6, wherein the input image is shot with a different exposure start time for each row of a plurality of rows and a different exposure duration for each pixel group of the plurality of pixel groups.

8. The imaging system according to claim 1, wherein the one or more processors are further configured to generate, from the plurality of exposure pattern images, a plurality of difference images.

9. The imaging system according to claim 8, wherein each difference image of the plurality of difference images comprises a plurality of pixels that have pixel values that are a difference between corresponding pixels of two or more of the plurality of exposure pattern images.

10. The imaging system according to claim 8, wherein the one or more processors are further configured to generate the corrected image based on synthesis of the plurality of difference images that have different exposure start times within the set interval.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a diagram describing a configuration of an imaging device and imaging processing;

(2) FIGS. 2A and 2B are diagrams describing a configuration of an imaging device and imaging processing;

(3) FIG. 3 is a diagram describing an imaging processing sequence by a focal-plane operation;

(4) FIG. 4 is a diagram describing an imaging processing sequence by a focal-plane operation;

(5) FIGS. 5A through 5D are diagrams describing an image distortion occurrence example as a problem with an imaging processing by a focal-plane operation;

(6) FIG. 6 is a diagram describing a configuration that reduces distortion by generating one image from multiple frames;

(7) FIG. 7 is a diagram describing a configuration example of an imaging apparatus serving as an image processing apparatus;

(8) FIGS. 8A and 8B are diagrams describing a setting example of an exposure pattern;

(9) FIG. 9 is a diagram describing a configuration of a distortion correcting unit and a processing example;

(10) FIG. 10 is a diagram describing a configuration of an intermediate image generating unit and a processing example;

(11) FIG. 11 is a diagram describing a specific example of processing executed by the intermediate image generating unit;

(12) FIGS. 12A and 12B are diagrams describing a specific example of processing executed by distortion correction processing unit;

(13) FIG. 13 is a diagram describing a specific example of processing executed by distortion correction processing unit;

(14) FIGS. 14A through 14C are diagrams describing an example of an exposure control pattern;

(15) FIG. 15 is a diagram describing an example of an exposure control pattern;

(16) FIG. 16 is a diagram describing an exposure control pattern and a setting example of exposure time and a timing image;

(17) FIG. 17 is a diagram describing a processing example of exposure time control;

(18) FIG. 18 is a diagram describing a configuration example of a distortion correcting unit;

(19) FIG. 19 is a diagram describing a distortion correction processing example that is executed with the configuration of the distortion correcting unit shown in FIG. 18;

(20) FIG. 20 is a diagram describing a configuration example of a distortion correcting unit;

(21) FIG. 21 is a diagram describing a distortion correction processing example that is executed with the configuration of the distortion correcting unit shown in FIG. 20;

(22) FIG. 22 is a diagram describing output processing of a high speed frame rate image that is executed by a distortion correcting unit that has no frame buffer;

(23) FIG. 23 is a diagram describing a configuration example of a distortion correcting unit;

(24) FIGS. 24A through 24C are diagrams describing an example of focal-plane distortion occurrence;

(25) FIGS. 25A through 25C are diagrams describing an image example applicable to correction processing of focal-plane distortion according to the related art;

(26) FIG. 26 is a diagram describing a correction processing example of focal-plane distortion according to the related art;

(27) FIG. 27 is a diagram describing an image example applicable to correction processing of focal-plane distortion according to an embodiment of the present disclosure;

(28) FIG. 28 is a diagram describing a correction processing example of focal-plane distortion according to an embodiment of the present disclosure; and

(29) FIG. 29 is a diagram illustrating examples of exposure control patterns.

DETAILED DESCRIPTION OF EMBODIMENTS

(30) Details of the image processing apparatus and image processing method and program according to the present disclosure will be described below with reference to the diagrams. Description will be given in the following order.

(31) 1. Configuration Example of Image Processing Apparatus

(32) 2. Details of Configuration and Processing of Distortion Correcting Unit

(33) 2-1. Overall Configuration and Processing of Distortion Correcting Unit

(34) 2-2. Processing of Intermediate Image Generating Unit

(35) 2-3. Processing of Distortion Correction Processing Unit

(36) 3. Other Embodiments

(37) 3-1. Modification of Exposure Control Pattern

(38) 3-2. Modification of Exposure Time Control

(39) 3-3. Modification of Frame Buffer Setting

(40) 3-4. Modification of Distortion Correcting Processing

(41) 4. Shared Configuration Example with Other Image Processing

(42) 5. Description of Advantages of Processing of the Present Disclosure

3. Configuration Example of Image Processing Apparatus

(43) First, a configuration example of the imaging apparatus serving as an embodiment of an image processing apparatus will be described with reference to FIG. 7. As shown in FIG. 7, the imaging apparatus has an optical lens 201, imaging device 202 that configured to have a CMOS image sensor or the like, distortion correcting unit 203 that corrects image distortion resulting primarily from focal-plane operations, signal processing unit 205, and control unit 207.

(44) The incident light enters the imaging device (CMOS image sensor) 203 via the optical lens 201. The imaging device 202 is an imaging device wherein sequential readout processing by region, e.g. by row, according to a focal-plane operation, is executed.

(45) The imaging device 202 accumulates a charge with the pixels of the imaging device according to the incident light, and outputs a photoelectric conversion signal based on the accumulated charge to the distortion correcting unit 203 as image data.

(46) The distortion correcting unit 203 inputs the output image signal from the imaging device 202, and corrects the image distortion resulting from a focal-plane operation. This processing will be described in detail later. The distortion correcting unit 203 outputs the corrected image 204 generated by distortion correction to the signal processing unit 205.

(47) At the signal processing unit 205, corrected image 204 that has been subjected to distortion correction is input, and predetermined image processing such as white balance adjusting processing, gamma correction processing and the like, for example, are performed on the input image, and an output image 206 is generated and output.

(48) The imaging apparatus is an imaging apparatus that can photograph moving or still images, and the image device (CMOS image sensor) 202 has a similar configuration as that described above with reference to FIG. 1. Note that the image device (CMOS image sensor) 202 is an imaging device that can control exposure time by region with the control of the control unit 207.

(49) For example, exposure time control by image region is executed with a certain configuration described in Japanese Unexamined Patent Application Publication Nos. 2006-253876 and 2006-542337, and Japanese Patent Application No. 2008-147818 described above, or another configuration according to the related art.

(50) Note that with the examples described below, as an example of the imaging device 202, an example using an imaging device having the configuration shown in FIGS. 8A and 8B will be described.

(51) The imaging device 202 has an RGB array as shown in FIG. 8A, and the exposure time (exposure patterns 1 through 4) that differs by units of four pixels is set, as shown in FIGS. 8A and 8B, as

(52) (a) Pixel block with the longest exposure time (Exposure Pattern 1)

(53) (b) Pixel block with the second longest exposure time (Exposure Pattern 2)

(54) (c) Pixel block with the third longest exposure time (Exposure Pattern 3), and

(55) (d) Pixel block with the shortest exposure time (Exposure Pattern 4).

(56) Pixel blocks with such four patterns of exposure times are set so as to be adjacent, as shown in FIG. 8A, and are cyclically arrayed. The exposure time ratio of the exposure patterns 1 through 4 is 4:3:2:1, and as shown in FIG. 8B, the exposure starting time differs by pattern, and readout times are equal.

2. Details of Configuration and Processing of Distortion Correcting Unit

(57) 2-1. Overall Configuration and Processing of Distortion Correcting Unit

(58) Next, the configuration and processing of the distortion correcting unit 203 will be described with reference to FIG. 9. As shown in FIG. 9, a sensor output image 211 which is an output image from the imaging device 202 is input into the distortion correcting unit 203. This sensor output image 211 is stored sequentially in the frame buffers 212 and 213. For example, when shooting a moving picture, the continuously shot frames are sequentially stored in the frame buffers 212 and 213.

(59) Note that processing can be performed to output one image, using a continuously shot image at time of still image photography also, and can be applied to the time of shooting of either still images or moving pictures. If we say that three continuously shot images are frames N−1, N, N+1, and the sensor output image 211 which is the newest shot image is frame N+1, then the frame images are stored, frame N in the frame buffer 212, and frame N−1 in the frame buffer 213.

(60) The intermediate image generating units 214 through 216 each generate multiple intermediate images as to the sensor output image 211 and the stored images in the frame buffers 212 and 213. This intermediate image generating processing will be described later.

(61) The multiple intermediate images generated by the intermediate image generating units 214 through 216 are input in the distortion correction processing unit 218. The distortion correction processing unit 218 inputs the multiple intermediate images generated by the intermediate image generating units 214 through 216, and further a scanning line position information 217 to be input from the control unit 207 is further input, the image distortion from a focal-plane shutter is corrected, and the corrected output image 219 is generated. This processing will be described later.

(62) 2-2. Processing of Intermediate Image Generating Unit

(63) Next, processing of the intermediate image generating unit configured within the distortion correcting unit 203 shown in FIG. 9 will be described with reference to FIG. 10. The distortion correcting unit 203 has three intermediate image generating units 214 through 216, as described with reference to FIG. 9.

(64) The intermediate image generating units 214 through 216 perform processing as to images that have each been consecutively shot. Specifically, the intermediate image generating unit 214 executes processing as to the sensor output image (frame N+1), the intermediate image generating unit 215 executes processing as to the stored image (frame N) in the frame buffer 212, and the intermediate image generating unit 216 executes the stored image (frame N−1) in the frame buffer 213.

(65) The processing to execute these three intermediate image generating units 214 through 216 differs only in the images to be processed, and basically are the same processing. Accordingly, the processing of the intermediate image generating unit 214 will be described below as a representative example.

(66) A sensor output image 211 is input into the intermediate image generating unit 214. The sensor output image 211 is an image having been shot, with exposure patterns (exposure patterns 1 through 4) that differ by pixel region, i.e., by setting four types of different exposure times, as described above with reference to FIGS. 8A and 8B.

(67) First, the interpolating processing unit 222 shown in FIG. 10 inputs the sensor output image 221 that has been set with four exposure times. The sensor output image 221 shown in FIG. 10 corresponds to the sensor output image 211 in FIG. 9. The interpolating processing unit 222 performs interpolation processing so that the exposure patterns are obtained for all of the pixels. That is to say, with each of the four exposure times for all of the pixels of the sensor output image 221, four exposure pattern images 223 through 226, which are equivalent to an image shot with exposure processing having been performed uniformly, are generated. Specifically, the four exposure pattern images below are generated:

(68) (a) Exposure pattern image 223 that is equivalent to an image shot with exposure pattern 1 having the longest exposure time,

(69) (b) Exposure pattern image 224 that is equivalent to an image shot with exposure pattern 2 having the second longest exposure time,

(70) (c) Exposure pattern image 225 that is equivalent to an image shot with exposure pattern 3 having the third longest exposure time, and

(71) (d) Exposure pattern image 226 that is equivalent to an image shot with exposure pattern 4 having the shortest exposure time; these four exposure pattern images are generated.

(72) The interpolation processing executed in exposure pattern image generating by the interpolation processing unit 222 is an interpolation method by a filter such as linear interpolation or the like, or a method wherein edge direction detection is performed and interpolation is based thereupon, or the like. The four exposure pattern images 223 through 226 generated by the interpolation processing unit 222 are each input into three different image generating units 227 through 229, with a pair having the least difference in exposure time as one pair.

(73) The processing of the difference image generating units 227 through 229 will be described with reference to FIG. 11. The exposure pattern images 223 through 226 that are equivalent to the images shot during the four different exposure times generated by the interpolation processing unit 222 shown in FIG. 10 are equivalent to the shot images during the four different exposure times (4T, 3T, 2T, 1T) as shown in FIG. 11.

(74) The difference image generating units 227 through 229 each input a pair having little difference in exposure time from the four exposure pattern images and compute difference pixel values of corresponding pixels, and generate the images made up of the difference pixel values thereof as timing images 230 through 232.

(75) The difference image generating unit 227 generates a first timing image 230 as below. The first timing image 230 is an image made up of the difference between a pixel value of the first exposure pattern image 223 which is equivalent to an image shot with the exposure pattern 1 which is the longest exposure time, and a pixel value of the second exposure pattern image 224 which is equivalent to an image shot with the exposure pattern 2 which is the second longest exposure time.

(76) The difference image generating unit 228 generates a second timing image 231 as below. The second timing image 231 is an image made up of the difference between a pixel value of the second exposure pattern image 224 which is equivalent to an image shot with the exposure pattern 2 which is the second longest exposure time, and a pixel value of the third exposure pattern image 225 which is equivalent to an image shot with the exposure pattern 3 which is the third longest exposure time.

(77) The difference image generating unit 229 generates a third timing image 232 as below. The third timing image 232 is an image made up of the difference between a pixel value of the third exposure pattern image 225 which is equivalent to an image shot with the exposure pattern 3 which is the third longest exposure time, and a pixel value of the fourth exposure pattern image 226 which is equivalent to an image shot with the exposure pattern 4 which is the shortest exposure time.

(78) Also, the fourth timing image 233 uses the fourth exposure pattern image 226 without change, which is equivalent to the image shot with the exposure pattern 4 which is the shortest exposure time. The four difference images correspond to images shot with the settings of the exposure times as shown in FIG. 11. That is to say,

(79) (1) First timing image 230 is an image shot at exposure time T (t0 to t1),

(80) (2) Second timing image 231 is an image shot at exposure time T (t1 to t2),

(81) (3) Third timing image 232 is an image shot at exposure time T (t2 to t3), and

(82) (4) Fourth timing image 233 is an image shot at exposure time T (t3 to t4).

(83) Thus, the four timing images (different images) 230 through 233 are equivalent to the four consecutively shot images at the same exposure time (T), wherein the shooting timing has shifted by T each time.

(84) Thus, each of the intermediate image generating units 214 through 216 shown in FIG. 9 generates four images shot at four different timings as to the input image. That is to say, the intermediate image generating unit 214 generates four timing images based on the sensor output image (frame N+1).

(85) The intermediate image generating unit 215 generates four timing images based on the stored image (frame N) of the frame buffer 212. The intermediate image generating unit 216 generates four timing images based on the stored image (frame N−1) of the frame buffer 213.

(86) 2-3. Processing of Distortion Correction Processing Unit

(87) Next, processing of the distortion correction processing unit 218 that is configured within the distortion correcting unit 203 shown in FIG. 9 will be described with reference to FIGS. 12A and 12B. The distortion correction processing unit 218 inputs a timing image generated by the intermediate image generating units 214 through 216, and corrects distortion caused by focal-plane shutter operations.

(88) The distortion correction processing here is described with reference to FIGS. 12A and 12B. FIG. 12A is a diagram showing exposure time of three consecutively shot images. The vertical axis represents rows and the horizontal axis represents time. As described above with reference to FIG. 6, in the case of performing shooting processing with a focal-plane operation, the exposure time shifts sequentially by row of the imaging device. FIG. 12A shows a setting example of the exposure time by row of the images of frames N−1, N, and N+1, which are three consecutively shot images.

(89) The images of frames N−1, N, and N+1 which are the three consecutively shot images correspond to images that are to be processed with the three intermediate image generating units 214 through 216 of the distortion correcting unit 203 shown in FIG. 9, i.e. correspond to the three images to be generated as timing images. That is to say, the intermediate image generating unit 214 generates four timing images based on the sensor output image (frame N+1).

(90) The intermediate image generating unit 215 generates four timing images based on the storage image (frame N) of the frame buffer 212. The intermediate image generating unit 216 generates four timing images based on the storage image (frame N−1) of the frame buffer 213.

(91) FIG. 12B is a diagram showing settings whereby the three intermediate image generating units 214 through 216 of the distortion correcting unit 203 can each distinguish four timing images generated based on the images of frames N+1 through N−1. As shown in FIG. 12B, the four timing images of frame N−1 are generated by the intermediate image generating unit 216 based on the stored image (frame N−1) of the frame buffer 213.

(92) The four timing images of frame N are generated by the intermediate image generating unit 215 based on the stored image (frame N−1) of the frame buffer 212. The four timing images of frame N+1 are generated by the intermediate image generating unit 214 based on the sensor output image (frame N+1).

(93) In FIG. 12A, three images (frame N−1 through frame N+1) are obtained by row, and in FIG. 12B, twelve (3×4=12) images (timing images) are obtained by row. The twelve images are equivalent to twelve images shot continuously which differ regarding exposure time (T). By generating the timing images in this way, data can be obtained with finer increments. With this fineness, reduction processing of distortion from a focal-plane shutter can be performed with higher precision that could be performed with the related art.

(94) An example of processing executed with the distortion correction processing unit 218 will be described with reference to FIG. 13. FIG. 13 is a diagram that is similar to the diagram shown in FIG. 12B. The diagram shows settings whereby the three intermediate image generating units 214 through 216 of the distortion correcting unit 203 can each distinguish four timing images generated based on the images of frames N+1 through N−1.

(95) For example, processing to generating an image with the output timing (Tx) shown in FIG. 13, i.e. a processing example in the case of generating an image shot wherein the exposure time is t1 through t2=Tx, will be described.

(96) Processing to compute the pixel values of the pixels on the row of the upper edge making up the image at the output timing (Tx) shown in this FIG. 13 will be described. In the event of image generating processing, timing images A, B, C, D, E shown in FIG. 13 will be used. Of the shooting timing of timing images B, C, D, all periods are included in a range of output timing (Tx) of the image planned to be generated. Accordingly, these timing images B, C, D are used without change.

(97) On the other hand, the shooting timing of timing images A and E is partially overlapping with the output timing (Tx) of the image planned to be generated and partially not overlapping. Accordingly, blending processing is performed which multiplies these by a predetermined weighting coefficient.

(98) The pixel values of the pixels on the row of the upper edge (OUT) making up the image at the output timing (Tx) shown in FIG. 13 are computed with the following (Expression 1)
OUT=a×A+B+C+D+(1−a)×E  (Expression 1)

(99) where A through E are pixel values at the same pixel position as the various timing images, i.e. at the corresponding pixel position, and a is a weighting coefficient.

(100) Note that the weighting coefficient a sets the value that is equivalent to the overlapping ratio of the output timing (Tx) shown in FIG. 13 and the various timing images, for example. Also note that the weighted coefficient a changes depending on the exposure timing of each row, so has to have a scanning line position information 217.

(101) In the above (Expression 1), the timing images B, C, D use the pixel values 100% without blending and without change, and so the blurring by interpolation processing can be minimized.

(102) As shown in FIG. 13, the timing images generated by the intermediate image generating unit 214 through 216 are images wherein the exposure time differing in increments of pixel rows is set, and the distortion correction processing unit 218 selects the timing image having the exposure time that is included in the exposure time of the corrected image planned to be generated in increments of pixel rows, and executes synthesizing processing to which the selected timing image is applied.

(103) Pixel value computing processing for rows on the upper edge has been described above, but for example in the case of computing pixel values of an image at output timing (Tx) shown in FIG. 13 for rows on the lower edge, processing using timing images P, Q, R, S that are shown in FIG. 13 is performed.

(104) The pixel value (OUT) of the pixel on the lower edge row can be found with (Expression 2) below
OUT=P+Q+R+S  (Expression 2)

(105) where P through S are pixel values at the same pixel position as the various timing images, i.e. the pixel values at corresponding pixel image positions.

(106) In this case, the total of the exposure timings of the four timing images exactly matches the output timing (Tx), and the timing image overlapping with the output timing (Tx) does not have to be used. Other rows can also similarly have the pixel values of the output image computed with pixel value adding processing of multiple timing images using the pixel value (OUT) computing expression.

(107) Thus, the distortion correction processing unit 218 synthesizes multiple timing images to compute the pixel values of the corrected images, and generates and outputs the corrected image 204.

(108) As we can see from the processing described with reference to FIG. 13 herein, the distortion correction processing unit 218 generates the corrected image 204 with the synthesizing processing of multiple images (timing images). The exposure time of the timing image is an image having ¼ the exposure time of the original shot image, and the corrected image 204 is generated with synthesizing processing that combines these. That is to say, from the upper edge row to the lower edge row, a corrected image 204 using a timing image that is included in roughly the same exposure time (Tx) can be generated.

(109) Consequently, the output image can be generated as an image shot with settings of roughly the same exposure time (Tx) for all of the rows from the upper edge row to the lower edge row.

(110) Accordingly, the output image generated with the present method has very little occurrence of image distortion resulting from focal-plane operations such as described above with reference to FIGS. 5A through 5D. The image distortion resulting from focal-plane operations results from shifts in exposure time between the upper edge row and lower edge row, but by performing synthesizing processing of the timing image shown in FIG. 13, an image wherein all of the rows from the upper edge row to the lower edge row are shot with the settings of approximately the same exposure time (Tx) can be generated.

(111) Consequently, the corrected image 204 generated by the distortion correcting unit 204 becomes an image that has suppressed image distortion.

(112) As shown in FIG. 7, the corrected image 204 output by the distortion correcting unit 203 subsequently is output to the signal processing unit 205 and subjected to predetermined signal processing such as white balance adjustments and y correction and so forth, then output as an output image 206.

3. Other Embodiments

(113) Next, other embodiments will be described.

(114) 3-1. Modification of Exposure Control Pattern

(115) In the above-described examples, an example is described wherein setting is performed for four exposure times in increments of pixel blocks of a rectangular region, with reference to FIGS. 8A and 8B as exposure control patterns of an imaging device.

(116) The setting patterns for exposure regions and exposure times are not restricted to such settings, and other various types of settings can be made. Regions to be control increments of exposure time of the input image may be set variously, such as pixel blocks made up of multiple pixels, or rows, or pixels.

(117) Note that in the above-described example, the number of exposure control patterns is four, but the present disclosure can be implemented if there are two or more patterns. An example of changing two exposure patterns is shown in FIGS. 14A through 14C. FIG. 14A is an example of changing the electronic shutter operation every two rows and alternately setting two exposure times in increments of two rows. FIG. 14B is an example of changing the electronic shutter operation every two by two pixels and setting the two exposure times every two by two pixels. FIG. 14C is a method of driving by changing the electronic shutter operation in further different increments. The example shows settings whereby three pixels in the vertical direction and three pixels in the horizontal direction are alternately selected and a region is set, and the two exposure times of the regions are alternately set. Setting various exposure patterns in this manner can be performed.

(118) Also, the pixel array described with reference to FIGS. 8A and 8B above and the pixel array shown in FIGS. 14A through 14C both have a pixel color filter in a Bayer array, but a pixel array that is for example a non-Bayer array of a pixel array as shown in FIG. 15 can also be applied.

(119) The pixel array shown in FIG. 15 is a configuration wherein the pixel array has W (white) pixels in addition to RGB. Note that in the case there are two exposure patterns, for example as shown in FIG. 16, the exposure time ratio of the exposure time of exposure pattern 1 and exposure time of the exposure pattern 2 can be set as 2:1. The following holds true for the distortion correcting unit 203, as shown in FIG. 16.

(120) (1) First timing image=exposure pattern 1−exposure pattern 2

(121) (2) Second timing image=exposure pattern 2

(122) These two timing images can be generated, and a corrected image that reduces the distortion by synthesizing processing of a timing image similar to the image generating processing described with reference to FIG. 13 above can be generated.

(123) 3-2. Modification of Exposure Time Control

(124) With the above-described example, a setting example wherein exposure starting is shifted in region increments, and the readout timings are coordinated, has been described as shooting processing in the case of obtaining imaging images having different exposure times, but the exposure time control by region increments can have various settings.

(125) For example, as shown in FIG. 17, the readout timing can be shifted to obtain the image. Additionally, with the exposure time control in region increments as described in Japanese Unexamined Patent Application Publication Nos. 2006-253876 and 2006-542337, and Japanese Patent Application No. 2008-147818 described above, or with other controls, an image can be shot with settings to periodically perform control of exposure time or readout timing, by row or by pixel, and processing of images shot with such various controls can be performed.

(126) 3-3. Modification of Frame Buffer Setting

(127) With the configuration described above with reference to FIG. 9, an example using two frame buffers is shown, but a configuration may use one frame buffer as shown in FIG. 18, and may use only two consecutively shot images. With this configuration, storage apparatus capacity and reading/writing bands can be reduced.

(128) Note that in the case of using this configuration, the distortion correction processing unit 218 generates a distortion corrected image using a timing image generated by applying only two consecutive frames as shown in FIG. 19, and not the processing applying the three frames of frame N−1 through frame N+1 as shown in FIG. 13.

(129) Further, the present disclosure may have a configuration wherein a frame buffer is not used, as shown in FIG. 20. Note that in the case of using this configuration, the distortion correction processing unit 218 generates a distortion corrected image using multiple timing images generated by applying only one frame as shown in FIG. 21, and not the processing applying the three frames of frame N−1 through frame N+1 as shown in FIG. 13.

(130) In this case, the setting of the exposure time period of an image output as shown in FIG. 21 results in a shorter time than the frame rate of the input image, whereby noise effects can occur. Also, as shown in FIG. 21, for example a timing image which includes exposure period of the data of the latter half portion of the output image timing for the upper edge row does not exist. Similarly, a timing image which includes exposure period of the data of the earlier half portion of the output timing for the lower edge row does not exist. Consequently, an image is generated just from a nearby timing image, whereby distortion reduction advantages by the focal-plane shutter operations can be thinned down slightly on the upper and lower edges. However, there is an advantage in that distortion can be corrected without using a frame buffer.

(131) Further, the distortion correcting unit 203 can also generate an signal of a frame rate that is higher speed than the frame rate of the input image. That is to say, the intermediate image generating unit generates a timing image as a consecutively shot image having an exposure time that is shorter than the input image, and makes up an output unit whereby the timing image generated by the intermediate image generating unit is output as a high frame rate image, thereby enabling a frame rate signal that is higher speed than the frame rate of the input image to be generated and output.

(132) Specifically, for example, as shown in FIG. 22, by generating an image with optional timing while performing correcting that eliminates focal-plane distortion as to the input image (frame N, N+1, . . . ), an image can be output that has a high frame rate of the output image (M, M+1, M+2, M+3, . . . ). Note that in the case of performing image output, performing predetermined image correcting such as brightness correcting and so forth and outputting is preferable.

(133) 3-4. Modification of Distortion Correction Processing

(134) With the examples described above, an example using FIG. 13 and (Expression 1) have been described as correction processing performed with the distortion correcting unit 203, but applying a correction method using a method other than such a linear interpolation may be used.

(135) For example, motion is detected from an image A and an image B which are two adjacent timing images shown in FIG. 13, and with motion compensation as to the image A, a motion compensation timing image A′ that is premised on being shot at the same timing as image B is generated. Similarly, motion is detected from an image D and an image E, and a motion compensation timing image E′ is generated.

(136) These images are used to compute the pixel value (OUT) of the corrected image, according to (Expression 2) shown below.
OUT=a×A′+B+C+D+(1−a)×E′  (2)

(137) where A′, B, C, D, E′ are pixel values at the same pixel positions of the various timing images or motion compensation timing image, i.e. of corresponding pixel positions, and a is a weighting coefficient.

(138) Note that the weighting coefficient a sets a value that is equivalent to a overlapping rate between the output timing (Tx) shown in FIG. 13 and the various timing images, for example. Also note that the weighting coefficient a changes depending on the exposure timing of the rows, so there has to be scanning line position information 217.

4. Shared Configuration Example with Other Image Processing

(139) The present disclosure includes a method to reduce distortion resulting from focal-plane shutter operations, but by using other processing also at the same time, image quality can be further improved. Several of such configuration examples will be described.

(140) In the example described above, in order to correct focal-plane distortion as shown in FIG. 10, processing is performed with the intermediate image generating unit to interpolate each pixel array of different exposure controls, and images are generated of the exposure patterns (first exposure pattern image 223 through fourth exposure pattern image 226) at all pixel positions.

(141) Such processing does not cause a problem for images that are blurred due to motion that causes distortion, but a problem occurs in that, when shooting a subject that is completely still, the resolution deteriorates.

(142) A configuration example of the distortion correcting unit 203 to reduce focal-plane distortion while preventing such resolution deterioration will be described with reference to FIG. 23.

(143) The sensor output image 211 is an image with exposure times that differ by pixel. A gain compensation processing unit 241 performs processing to multiply the gain according to exposure time in region increments of the sensor output image 211 by the pixel values.

(144) For a still image, an image having no resolution deterioration that is the same as the normal Bayer array can be obtained by executing the gain compensation processing. The output of the gain compensation processing unit 241 has distortion, and blurring amounts of the motion for each exposure control pattern differs, whereby in the case there is motion, image breakdown occurs.

(145) Therefore, motion is detected by pixel or by area with the motion detecting unit 242, and with a motion adapting processing unit 243, selecting processing is performed such that an output image of the distortion correction processing unit 218 is used in a pixel region having motion, and an output image of the gain compensation processing unit 241 is used in a location having no motion, or blending processing is performed according to the motion amount. Thus, the resolution of an image having no motion is as it has been in the past, and locations having motion can reduce the distortion.

(146) Further, portions having no motion use a frame memory to perform noise reduction processing in the temporal direction (also called three-dimensional NR or 3DNR), thereby reducing noise in the still image portions. Further, a configuration may be made which adds a pixel value saturation countermeasure processing.

(147) In FIG. 11 described above, in order to generate the timing images 230 through 233, the difference in exposure pattern images that differ in the difference image generating units 227 through 229 is calculated, but for example in the case that the pixel value of one of the images to be subjected to difference calculation is saturated, an accurate difference value is not be obtained.

(148) For example, in the case that the corresponding pixel values of the two images applied to generate the timing image (difference image) are originally 1200 and 800,
Difference image pixel value=1200−800=400

(149) holds true. However, in the case that the output of the sensor is 10 bit, the sensor output can only output the pixel values (0 through 1023). In this case the pixel value 2000 mentioned above is output as pixel value 1023,
Difference image pixel value=1023−800=223

(150) holds true, and a timing image (difference image) having a pixel value smaller than the actual can be generated.

(151) As a saturation countermeasure, the simplest is to perform clipping processing with values that differ according to exposure patterns.

(152) As shown in FIG. 11, the exposure ratio of the exposure patterns 1, 2, 3, 4 is 4:3:2:1, and in the case the pixel value output of the imaging device is 10 bit, after clipping the pixel in exposure pattern 1 with 1023, the pixel in exposure pattern 2 with 768, the pixel in exposure pattern 3 with 512, and the pixel in exposure pattern 4 with 256, the difference image generating can be performed, whereby the problem of saturation can be resolved.

(153) Also, with another method, saturation is detected, and a difference image is not generated for the saturated portion. For this saturated portion, pixel value setting is performed by a dynamic range expanding method that is disclosed in PTL 3, for example. With such processing, for example, the distortion reduction effect is weakened, but the dynamic range can be expanded.

5. Description of Advantages of Processing of the Present Disclosure

(154) Next, advantages of processing according to the present disclosure will be described.

(155) In order to show the advantages of the present disclosure, focal-plane distortion will be described with reference to FIGS. 24A through 24C. FIG. 24A is an image example at the time that a still object is imaged, and FIGS. 24B and C are image examples at the time that an object moving in the horizontal direction from left to right is imaged. FIG. 24B is an image example at the time of imaging with a global shutter (wherein focal-plane distortion does not occur), and FIG. 24C is an image example at the time of imaging with a focal-plane shutter. As shown in FIG. 24C, object distortion occurs with a focal-plane shutter operation.

(156) As a comparison with the related art, a focal-plane distortion reduction effect with a method shown in Japanese Unexamined Patent Application Publication No. 2006-148496 (or FIG. 12A) will be shown. In the method in Japanese Unexamined Patent Application Publication No. 2006-148496, the three images shown in FIGS. 25A through 25C are used to reduce the focal-plane distortion. The difference in readout time by row is taken into consideration, a blend coefficient is computed, and one image is generated from three images.

(157) The distortion reduction processing result from the Japanese Unexamined Patent Application Publication No. 2006-148496 is shown in FIG. 26. In FIG. 26, the position of the center of gravity of the object is adjusted, but the upper and lower edges of the screen can blur greatly, and the overall reduction effect is not great.

(158) The focal-plane distortion reduction effect according to the present disclosure will be shown. As described with reference to FIG. 11, FIG. 12B, and FIG. 13, four exposure pattern images are generated first from one image made up of four exposure times, and four timing images equivalent to images shot with four different timings are generated as a difference image of four further exposure pattern images. The distortion correction processing unit generates a corrected image wherein the focal-plane distortion is corrected by a synthesizing processing of these timing images.

(159) For example, a specific example of multiple timing images (difference images) is an image such as that shown in FIG. 27. The timing images are generated as difference images of multiple exposure pattern images, whereby the exposure period for each of the timing images is a short period, and a dark image is obtained, but many images with little blurring and a short time spacing can be obtained. The corrected image made up of the pixel values computed based on the image synthesizing processing using the image in FIG. 27, e.g. the pixel value computing processing above with reference to FIG. 13 becomes an image such as that shown in FIG. 28. That is to say, by performing reduction processing of focal-plane distortion, FIG. 28 can be obtained.

(160) As compared to the image shown in FIG. 26 which is the result of a method of related art, the image shown in FIG. 28 which is the result of the present disclosure can be confirmed as having an extremely small amount of irregularities in distortion reduction by the position on the screen, and as having a great distortion reduction effect.

(161) As shown above, the image set at exposure times that differ by region is shot with an imaging device, according to the present disclosure, and with the processing applying the shot image, images that have performed focal-plane distortion correcting can be generated.

(162) In order to obtain a similar effect with a method according to related art, a sensor has to operate at a high speed, but not with the present disclosure, so demerits such as increased power consumption that accompany an increase in operation speed of the imaging device do not occur. Also, for a similar reason, the storage apparatus serving as a frame buffer does not have to be capable of high speed operations, and power consumption and apparatus cost can be reduced.

(163) Further, complicated computations such as motion vector calculating processing and the like do not have to be performed, so reduction of computing load and high speed processing are realized. Note that even in the case wherein the present disclosure and the motion vector are configured so as to share the computing processing, the time difference between timing images that are to be subjected to motion vector computing is small, and motion vector computing processing can be executed with high precision, whereby a highly precise distortion reduction effect can be obtained.

(164) The present disclosure have been described above with reference to specific embodiments. However, it is obvious that one skilled in the art can make modifications and substitutions to these examples within the scope and essence of the present disclosure. That is to say, the present embodiments have been disclosed in exemplary form, and should not be interpreted in a restricted manner. In order to determine the essence of the present disclosure, the Claims should be referenced.

(165) Also, the series of processing described in the Specification can be executed with hardware, software, or a combined configuration of both. In the case of executing a processing with software, a program having recorded a processing sequence is installed in the memory of a computer that is built in to dedicated hardware, and processing is executed, or a program is installed in a general-use computer that can execute various types of processing, and the processing is executed. For example, the program can be recorded beforehand in a recording medium. In addition to being installed from a recording medium to a computer, the program can be received via a network such as LAN (Local Area Network) or the Internet, and installed in a recording medium such as a built-in hard disk.

(166) Note that the various types of processing described in the Specification are not only executed in a time-series manner according to the description, but may be executed in parallel or individually, according to the processing capability of the apparatus executing the processing, or as suitable. Also, a system according to the present Specification is a theoretical collective configuration of multiple apparatuses, and is not restricted to apparatuses with various configurations within the same housing.

(167) The present disclosure contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2011-038240 filed in the Japan Patent Office on Feb. 24, 2011, the entire contents of which are hereby incorporated by reference.

(168) It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.